Voice vector framework for authenticating user interactions

ABSTRACT

There are provided systems and methods for a voice vector framework that authenticates user interactions. A service provider server receives user interaction data having audio data that is associated with an interaction between a user device and the service provider server. The server extracts user attributes from the audio data and obtains user account information associated with the user device. The server selects a classifier that corresponds to a select combination of features based on the user account information and applies the classifier to the user attributes. The server generates a voice vector that includes multiple scores indicating likelihoods that a respective user attribute corresponds to an attribute of the select combination of features. The server compares the voice vector to a baseline vector corresponding to a predetermined combination of features and sends a notification to an agent device with an indication of whether the user device is verified.

TECHNICAL FIELD

The present application generally relates to machine learning modelstrained for authenticating user interactions and more particularly to anengine having a machine learning model trained to authenticate userinteractions with a voice vector framework, according to variousimplementations.

BACKGROUND

Electronic service providers may provide an online marketplaceenvironment for users, which may be used to buy and sell goods withother entities. Some of these services may be used maliciously byfraudulent users, such as overtaking a merchant account and takingcontrol of payments made for goods sold, which poses a significant riskto these service providers. Since the service providers may processthousands (or more) transactions daily, it may be difficult to reviewwithout a large review and compliance team.

Moreover, tactics in performing prohibited transactions electronicallyare ever-evolving and becoming more sophisticated. Electronic serviceproviders need to keep pace with the fraudulent users in providingsecurity measures, such as accurately evaluating risk (e.g., detectingprohibited transactions) in real-time. In this regard, computer modelsare often utilized to assist in evaluating risks of electronictransactions.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a block diagram of a networked system suitable forimplementing the processes described herein, according to animplementation of the present disclosure;

FIG. 2 illustrates a block diagram of an event detection server,according to an implementation of the present disclosure;

FIG. 3 is a flowchart of an example process of actionable insightanalysis into user interaction data, according to an implementation ofthe present disclosure;

FIG. 4 is a flowchart of an example process of probabilistic anomalydetection and mediation, according to an implementation of the presentdisclosure;

FIG. 5 illustrates a block diagram of a networked system for automateddevice data retrieval and analysis, according to an implementation ofthe present disclosure;

FIG. 6 conceptually illustrates an exemplary workflow of the automateddevice data retrieval and analysis of FIG. 5 , according to animplementation of the present disclosure;

FIG. 7 is a flowchart of an example process of automated device dataretrieval and analysis, according to an implementation of the presentdisclosure;

FIG. 8 conceptually illustrates an exemplary workflow of a voice vectorframework for authenticating user interactions, according to animplementation of the present disclosure;

FIG. 9 is a flowchart of an example process of a voice vector frameworkfor verifying user interactions, according to an implementation of thepresent disclosure;

FIG. 10 is a flowchart of an example process of a voice vector frameworkfor detecting malicious activity in user interactions, according to animplementation of the present disclosure; and

FIG. 11 is a block diagram of a computer system suitable forimplementing one or more components in FIG. 1 and FIG. 2 , according toan implementation.

Implementations of the present disclosure and their advantages are bestunderstood by referring to the detailed description that follows. Itshould be appreciated that like reference numerals are used to identifylike elements illustrated in one or more of the figures, whereinshowings therein are for purposes of illustrating implementations of thepresent disclosure and not for purposes of limiting the same.

DETAILED DESCRIPTION

An online service provider that provides electronic services, such aselectronic content access, electronic transactions, etc. may offernumerous avenues for users to interact with the online service provider.For example, users may interact with the online service provider byaccessing a website or a mobile application associated with the onlineservice provider. In another example, the user may communicate with theonline service provider by calling a hotline associated with the onlineservice provider, sending emails to a designated email addressassociated with the online service provider, and/or chatting with anagent (or a chatbot) of the online service provider via a chatapplication. These various avenues provide convenient access to theonline service provider, but also offer opportunities for malicioususers to perform fraudulent access of data and/or service of the onlineservice provider. Thus, various aspects of the disclosure, as discussedin detail below, provide frameworks and platforms for efficientlydetecting malicious activities during the interactions with the onlineservice provider and performing actions to reduce loses caused by themalicious activities.

Probabilistic Anomaly Detection in Streaming Device Data

It is often difficult to identify unusual data when viewing steamingdevice data (e.g., reviewing logs attempting to find fraudulentinformation). The streaming device data may be associated with deviceattributes associated with devices that have attempted connection withan online service provider (e.g., an online server). The devices mayattempt to access electronic services offered by the online serviceprovider (e.g., logging in to a user account with the online serviceprovider, accessing data from the online service provider, initiating anelectronic transaction such as an electronic payment transaction througha user account with the online service provider, etc.) When reviewingthe data manually, it is not easy or intuitive to identity anomalies.For example, when looking through events that contain differentdescriptors for mobile devices, it is difficult to determine which logentries represent prohibited transactions (e.g.,fraudulent/malicious/falsified data).

Thus, according to an aspect of the disclosure, an anomaly detectionsystem provides an anomaly detection framework that identifies anomaliesduring interactions of the online service provider with a user device inreal time based on device attributes associated with the user device. Insome embodiments, the anomaly detection system may leverage thelikelihood of a given combination of device attributes in a dataset.Upon detecting an anomaly during the interactions with the user device,the anomaly detection system may automatically take one or more actions,including but not limited to, alerting an agent, providing a differedexperience for a user of the user device, re-securing a user accountthat is being accessed by the user device, etc. For example, the anomalydetection system may provide additional restrictions to data and/orservices that can be accessed by the user device.

Devices of the same type (e.g., same manufacturers, same models, etc.)may share similar qualities (e.g., attributes). Devices of a particulartype can look similar to each other when viewed via a log file. Thissimilarity also extends to software platforms. For example, if asmartphone of a particular brand and a particular model version (e.g.,Apple® iPhone 8) has a specific screen height and width, then everysmartphone of the same model version that connects to a service providersystem should have features that align with others using that same modelversion. If, however, the anomaly detection system detects that the userdevice attempted to connect to the service provider system (e.g., toaccess the website of the service provider system, to access a useraccount with the service provider system, to perform an electronictransaction through a user account with the service provider system,etc.) has features that differed greatly from those of its same type,the anomaly detection system may determine that an anomaly is presentand should be investigated. By leveraging a statistical analysistechnique with machine learning, such as a probability density function(PDF), among others, the anomaly detection system may isolate anomaliesby holding some device attribute values constant and generating machinelearning-trained statistical models around the remaining deviceattributes. The anomaly detection system may identify outliers andanomalies in real time over streaming device data, and automaticallytake a remedial action based on desired outcomes.

Existing approaches in anomaly detection can acquire a set of data, plotthat data, and attempt to visualize what is different among the rendereddata. In contrast, the anomaly detection system of some embodimentsleverages machine learning to identify fraudulent and/or suspiciousdevices attempting to connect with the service provider system in realtime via their device attributes in an effort to automatically invoke aprohibited transaction. The anomaly detection framework disclosed hereinis more cost efficient than existing anomaly detection systems andflexible enough to be useful for service providers and/or usersattempting to defend systems from bad actors.

In a first use case example, a research individual is attempting toidentify malicious activity on an online payment processing system. Theresearch individual generates logs that cover a specified time-period ofsuspected prohibited activity. The log contains thousands, if notmillions, of lines of data. The research individual first tries to plotfeatures to find correlations and analyze the data. The researchindividual plots histograms and attempts clustering techniques, but tono avail. The research individual then leverages the anomaly detectionsystem to isolate device features and generate multi-dimensionalprobability maps of the remaining device features. These probabilitymaps help to identify outliers within the data. The research individualnow has a much smaller subset of data from which to identify themalicious activity within the system.

In a second use case example, devices of a certain type can have certainfeatures that are static (e.g. their model number). As such, the anomalydetection system may use that information to hold that value constantand generate a dynamic density map of interactions with the remainingdevice features by leveraging a subset of last-known instances of thatdevice. Based on the density map, the subject technology can identifyand detect which device-to-system connections have a minimal (or lowprobability) of occurrence given the other constant values. Thoseconnections can then be acted upon with the following outcomes: (1) theconnection attempt can be automatically directed to a secondary flow forfurther analysis, (2) the connection attempt can be stepped up withadditional user authentication prompts, (3) an agent can be alerted toan anomalous connection, and/or (4) additionally security checks can beautomatically performed to validate the authenticity of the connection(e.g., vendor identifier (VID) lookup, account review, etc.).

Analysis Platform for Actionable Insight into User Interaction Data

The online service provider that provides electronic transactionservices may receive a large amount of communication attempts fromlegitimate users and malicious users, which include emails that aretransmitted to a dedicated e-mail address associated with the onlineservice provider, phone calls received via a support hotline associatedwith the online service provider, chat sessions initiated via a websiteof the online service provider, etc. These e-mails, calls, and chatinteractions with the online service provider can be collectivelyreferred to as customer inputs. Many of the customer inputs may be fromlegitimate users who are requesting help for resolving issues related totheir user accounts (e.g., disputing a transaction, issues with logginginto an account, etc.). However, at least some of the customer inputs isnot related to legitimate users' concerns (e.g., subscriptions,advertisements, etc.), and some may even include malicious content(e.g., malware attachments, phishing emails or phone calls, etc.).

Processing through such a high volume of customer inputs manually is notfeasible both in terms of resources and time. Thus, according to anotheraspect of the disclosure, an analysis system may be configured toautomatically classify and analyze customer inputs (e.g., email, phone,chat, etc.) for producing actionable insights related to fraudcampaigns, phishing attacks, malware distribution, or product issues.

In some embodiments, the analysis system may classify customer inputs(e.g., customer complaints and/or issues reported by e-mails, voicecalls, and/or chat sessions) into respective categories for automatedanalysis and remedial action. The analysis system may classify acustomer input into one or more fraud categories based on matching thecustomer input to one or more known (or stored) fraud patterns. Theanalysis system may also identify new fraud campaigns not yet known. Insome embodiments, the analysis system may detect common victim patternsin user accounts that have been exposed to fraud. Based on the detectedpatterns, the analysis system may invoke risk rules to preventfraudulent activities associated with the user account. For example, theanalysis system may detect that a customer input is associated with aphishing scheme. The analysis system may then forward the customer inputto a phishing sub-module for analysis and action. In another example,the analysis system may detect whether a customer input (e.g., an email)includes an attachment (e.g., a data file) that is malicious. Theanalysis system may then generate a signature for the attachment and mayblock the customer input from being presented in an email inbox.

In some embodiments, upon receiving a customer input, the analysissystem may classify the customer input into one of multiple categories.Example categories may include: (1) fraud complaints, (2) phishing, (3)malware, (4) other, and/or (5) noise. The analysis system may includeadditional categories for classification to accommodate larger datasetsof customer input. For classification, the analysis system may includeor use a machine learning model, such as support vector machines (SVMs)or Random forest classifiers that are trained on previously labeledcustomer input for each category. The labeled dataset can be gatheredfrom customer support agents. For feature extraction, the analysissystem may utilize different representations including bag-of-words,term frequency inverse document frequency (TF-IDF), document to vectorrepresentation (Doc2Vec) that uses a deep learning approach. Theanalysis system may extract topics using Latent Dirichlet Allocation(LDA), and select the best performing feature extraction model toextract features from the customer inputs. In some embodiments, theanalysis system may also augment the model with information regardingthe caller/sender of the customer input. Once a customer input isclassified in one of these categories, further action can be taken oneach interaction.

When the customer input is classified as a fraud complaint, the analysissystem may cluster the customer input with other similar reported fraudactivities (e.g., other customer inputs that have been classified as thesame category). For example, a user may contact the online serviceprovider to complain about receiving invoices related to renewing awebsite domain from a domain provider. When the analysis systemclassifies such a customer input as a fraud complaint, the analysissystem may cluster such a customer input with other related complaintsover invoices associated with renewing website domains. In someembodiments, since certain complaints may relate to multiple types offrauds, the analysis system may implement a probabilistic clusteringtechnique, such as Gaussian mixture model (GMM). GMM can assign aprobability to each fraud complaint based on which cluster is assignedto the fraud complaint. If a complaint does not seem to fall in any oneof the existing clusters, a new cluster can be created. Once a clusterreaches a certain number of complaints, a report generation model can betriggered to generate a report. The analysis system may identifyspecific information from the complaints, such as email addresses of thecustomers or any transaction details, and derive patterns from thecomplaints that belong the same cluster such as country of origin of thecustomers, customer age range, network addresses used for thetransaction, transaction amount that was charged, a description of theservice if any, and so on. This information can be added to the reportwith all the insights gathered and forwarded to an agent device forfurther analysis and action. As such, the analysis system may increasethe efficiency of handling customer inputs (e.g., legitimate customerinputs may be forwarded to the right personnel or chatbots to handle),while malicious customer inputs are properly classified and forwarded todifferent modules for further analysis and actions.

In an example use case, the online service provider may receive usercomplaints in the form of calls, e-mails, and/or chat regarding invoicessent to them to pay for a web site domain renewal by a web hostingcompany. With a clustering technique, these complaints can be groupedtogether based on the content of the e-mail, the amount requested, andthe hosting provider. Once a predetermined number of complaints (e.g.,exceeding a threshold) is identified, the analysis system may betriggered to generate a report. The analysis system may analyze thegrouped customer inputs (e.g., within the same cluster) to derivepatterns. For example, the analysis system may determine that a majorityof the customers associated with the grouped customer inputs fall in aspecified age band (e.g., older than 50 years of age) and are all knownto reside in the United States. The invoices are determined to originatefrom accounts registered in Russia where the hosting provider has noplace of business or business presence. The analysis system maydetermine a risk level based on the derived patterns, and may alert arisk team to investigate the complaints in more detail when the riskexceeds a threshold. In some embodiments, the analysis system may alsoperform actions such as restricting access to the user accountsassociated with the complaints when the risk level exceeds thethreshold.

Automated Device Data Retrieval and Analysis Platform

According to another aspect of the disclosure, a browser analysis systemof the online service provider may employ anti-fraud defense techniquesto generate fingerprints for various types of web applications (e.g.,browsers or other web clients) connecting to the online serviceprovider. In some embodiments, upon detecting an attempt by a webapplication of a user device to connect to the online service provider,the browser analysis system may transmit code to the user device andcause the user device to execute the code. The code, when executed bythe web browser, is configured to examine webpage attributes (e.g.,document object model (DOM) attributes) and/or values of a webpage beingloaded and presented on the user device. The webpage attributes and/orvalues may enable the browser analysis system to a) track a unique userover time, and b) determine an anomaly, such as whether the webapplication is providing inaccurate information of its actual identitythrough malicious manipulation of the web browser attributes and values.

To determine which attributes, behaviors, values, etc., are relevant fordetecting anomalies is a time-consuming and daunting task that requiresskilled developers to manually examine each individual type of webapplications. Since the attributes, behavior, and/or values that arerelevant for detecting anomalies for one type of web applications maynot be relevant for another type of web applications, the skilleddevelopers may be required to perform this manual examination for eachtype and each version of web applications, and may have to continue toperform manual examination when a new version of a browser is released.Otherwise, the existing device data can lose effectiveness over time asthe web browser changes and evolves.

Devices of the same type (e.g., same manufacturers, same models, etc.)may share similar qualities (e.g., attributes). Devices of a particulartype can look similar to each other when viewed via a log file. Thissimilarity also extends to software platforms. For example, if asmartphone of a particular brand and a particular model version (e.g.,Apple® iPhone 8) has a specific screen height and width, then everysmartphone of the same model version that connects to a service providersystem should have features that align with others using that same modelversion. If, however, the online platform detects that the user deviceattempted to connect to the service provider system (e.g., to access thewebsite of the service provider system, to access a user account withthe service provider system, to perform an electronic transactionthrough a user account with the service provider system, etc.) hasfeatures that differed greatly from those of its same type, the onlineplatform may determine that an anomaly is present and should beinvestigated.

In some embodiments, the browser analysis system provides in-depth andautomatic testing of current and pre-release web applications with theintent of generating a comprehensive database of their behaviors,features, and functionality in order to provide additional capabilitiesfor detection and mitigation of malicious actors using modified orspoofed client devices. The browser analysis system may also allow baselining for statistical modeling of browser traffic and can generatealert notifications of new functionality that can be implemented intoanti-fraud defense mechanisms.

Voice Vector Framework for Authenticating User Interactions

Fraudulent calls are often made to customer service agents with the goalof lifting restrictions on an account or performing account takeoverusing password reset initiated by the customer service agent. Socialengineering techniques are often employed where the caller providesenough information to convince the agent to perform the desired actionon an account that is not theirs. Additionally, these types of attacksoccur on a scale where the same person may be calling for multipleaccounts.

As such, in another aspect of the disclosure, a voice authenticationsystem may detect fraudulent calls based on analyzing voicecharacteristics of the callers. In some embodiments, using various voiceanalysis techniques disclosed herein, the voice authentication systemmay authenticate a caller as a legitimate user associated with a useraccount of the online service provider and also determine if the callerhas called before on multiple other user accounts of the online serviceprovider. When the voice authentication system has determined that thecaller is suspicious (e.g., that the caller is not a legitimate userassociated with the user account and/or that the caller is linked toprevious call(s) associated with other user account), the voiceauthentication system may provide real-time feedback by alerting anautomated chat module (e.g., a chatbot) or customer service agents ofthe online service provider of such suspicious activity. In someembodiments, the voice authentication system may leverage variousaspects of speech recognition and voice identification technology aswell as intent identification on the incoming customer call. The voiceauthentication system may provide a framework for a two-stage procedurethat first verifies the identity of a caller and second to check if thesame caller has previously called with the same intention on a differentaccount not belonging to him or her. This framework can identifyfraudsters, generate a voice blacklist, and alert customer serviceagents in real time to mitigate any security lapse.

In order to facilitate analyses of voice characteristics of incomingcallers, the voice authentication system may generate multiple machinelearning-based voice models that represent variations of voicesassociated with different user accounts of the online service provider.In some embodiments, the voice authentication system may generatemultiple generic voice models, where each of the generic voice modelsmay correspond to a particular cross-section of demographics. Examplesof the demographics may include gender, country of origin (e.g.,accent), age, among others. An example cross-section can be a young male(e.g., younger than 30 years old) from France, or an elderly female(e.g., older than 60 years old) from New York. There can be a number ofgeneric voice models generated to represent individual attributes andselected cross-sections that together encompass the widest possiblegroup of individuals. In some embodiments, the voice authenticationsystem may generate the machine learning-based generic voice modelsbased on different combinations of a particular subset of thedemographics attributes, such as gender, age, and accent. The machinelearning-based voice models may be configured to identify age groups ina binary manner such as old/young, or categorically by dividing thedifferent ages into different age groups.

The voice authentication system may select, for each user account, aparticular machine learning-based generic voice model based on voicecharacteristics of a user of the user account. Thus, the voiceauthentication system may select, for a user account associated with afemale user who is 26 years old from France, a generic voice modelcorresponding to a female-young-French voice. In another example, thevoice authentication system may select, for a user account associatedwith a male user who is 65 years old from England, a generic voice modelcorresponding to a male-old-English voice.

The generic voice model can be used by the voice authentication systemto generate multiple different voice models specific to each of the useraccounts. For example, the voice authentication system may train, for aparticular user account, the generic voice model using different audiofiles associated with the particular user account to generate differentvoice models corresponding to different call intentions. The audio filesassociated with the user account may be obtained based on historicalcalls as previously identified by customer service agents (e.g.,verifying that the caller was, in fact, the rightful owner of the useraccount). In some embodiments, the voice authentication system mayidentify different audio files associated with a user accountcorresponding to different call intentions. In general, the onlineservice provider may determine one or more call intentions of thecallers based on the type of services provided by the online serviceprovider. Common call intentions may include “password reset,” “paymenttransaction,” “payment dispute,” or other types of intentions. The voiceauthentication system may categorize the audio files associated with theuser account based on the different call intentions. The voiceauthentication system may then extract keywords from the audio filesthat correspond to each of the call intentions. For example, for thecall intention of “password reset,” the voice authentication system mayextract, from the audio files corresponding to the “password reset” callintention, phrases such as “password reset,” “resetting my password,”“password resetting,” etc. These audio files with the extracted keywordscan be grouped together for training a “password reset” model for theuser account.

The voice authentication system may generate and train a voice model,for the user account and a corresponding call intention, based on theextracted keywords from the audio files using the generic voice model.Thus, the voice authentication system may generate, for each useraccount based on the corresponding generic voice models, multiple voicemodels for the different call intentions. Since each of the voice modelsis trained with the same phrase (or multiple similar phrases), anyvariation within each of the resulting voice models can be due to audioquality and patterns of speech. Pre-processing can be performed on theaudio files (having the extracted keyword) to normalize the variationand eliminate noise. The result can be a set of trained voice modelswhose variation would be the result of the differences in speechpatterns from one group to the other. The machine learning-based voicemodels can be periodically updated using previously saved and taggeddata.

In the first stage of the framework, when the voice authenticationsystem detects an incoming call inquiring about a user account with theonline service provider, the voice authentication system may determinewhether the caller is associated with the user account based on thegenerated voice models associated with the user account. The incomingcall may include voice data associated with phrases and utterances madeby the caller. In some embodiments, the voice authentication system mayidentify a call intent of the incoming call based on the voice data. Forexample, the voice authentication system may extract one or morekeywords from the voice data, and classify the call as one of themultiple call intentions based on the extracted keywords. In oneexample, if the voice authentication system detects a phrase that islinked to one of the voice models (e.g., the phrase “resetting mypassword”) in the voice data, the voice authentication system mayclassify the call as the call intention associated with the voice model(e.g., the “password reset” call intention). The voice authenticationsystem may then select the voice model that corresponds to the extractedkeyword(s) to authenticate the caller.

In some embodiments, upon classifying the call as a particular callintention, the voice authentication system may determine whether thecaller is a legitimate user of the user account using a voice modelassociated with the user account and corresponding to the callintention. When comparing the caller's audio sample (e.g., thekeyword(s) extracted from the voice data) against the selected voicemodel, the voice authentication system or the voice model may generatean “accuracy” or “confidence” score. In some embodiments, the voicemodel may output a value (e.g., a confidence score) indicating how closethe audio sample is to a voice of the same phrase generated by themodel.

In some embodiments, the voice authentication system and/or the voicemodels may determine the confidence score based on voice vectorsassociated with the voice data of the caller. For example, for eachvoice model generated for the user account, the voice authenticationsystem may generate a signature in the form of a vector that representsthe attributes of the voice represented by the machine learning-basedvoice model. The attributes can be generated using a combination ofselective phrases and individual words. Within the voice authenticationsystem that records the calls, the phrases and words can be segmentedand stored as tagged data. For example, someone saying “password reset”can be a key phrase that is captured. Using methods such as HiddenMarkov model, gaussian mixture model, deep learning models, amongothers, the voice authentication system may generate a vectorrepresentation and use this vector for comparison. The vector comparisoncould be accomplished using a vector similarity metric. An example ofsuch a metric would be a multi-dimensional relative distance metric,such as leveraging an ‘n’-dimensional Euclidean distance calculation togenerate a per-dimension distance as well as an overall distance. Thetechnique for measuring the distance would depend on the underlyingtechnique used to generate the voice representation. By examining therelative distance of an incoming sample to existing ones, the voiceauthentication system would generate a confidence score.

In one example, upon detecting the incoming call, the voiceauthentication system may generate a voice vector based on the voicedata of the incoming call. The voice authentication system may thencompare the voice vector associated with the voice data of the callerwith the voice vector generated for the voice model using the techniquesdisclosed herein. The voice authentication system and/or the voice modelmay determine a confidence score based on a similarity between the twovoice vectors. If the confidence score is above a threshold, then thevoice authentication system may consider the voice a match to the model,and may determine that the caller is a legitimate user of the useraccount. For example, the voice authentication system may determine thatthe caller is the legitimate user of the user account when theconfidence score is above a threshold, and may determine that the calleris not the legitimate user of the user account when the confidence scoreis below the threshold. Once the caller has been verified, theconfidence score can be leveraged as a way to gauge any drift that hasoccurred in an individual's speech over time as a way of improving themodel for the user account. Furthermore, regardless whether the calleris determined to be a legitimate user of the user account, the voicedata (e.g., the voice vector) may be stored in association of the useraccount, to indicate that such a caller has called regarding the useraccount. In addition, if the caller's voice data is determined to be nota legitimate user of the user account, the voice data may be classifiedinto one of the generic voice model by comparing the voice data againsteach of the generic voice model.

In the second stage of the framework, the voice authentication systemmay use the voice data to determine whether the caller has previouslycalled for another user account. In some embodiments, the voiceauthentication system may compare voice data (and/or the voice vectordetermined based on the voice data) against different voice modelsassociated with different user accounts. For example, the voiceauthentication system may first compare the voice data (and/or the voicevector) against the different generic voice models to determine whichgeneric voice model matches the caller. The voice authentication systemmay determine a confidence score for each generic voice model indicatinghow well the voice data of the caller matches the generic voice model.This confidence score can determine which model more closely representsthe caller.

For example, a young male with a French accent would yield a highconfidence score on the male-young-French-accent voice model but yield amuch lower confidence score on the female-old-Russian model. Once thecaller's voice matches with one of the generic voice models based on acertain threshold, individual voice vectors from the voice modelsspecific to various user accounts, that were derived from the matchedgeneric voice model, and various voice vectors associated with thematched generic voice model can be used to determine if the voicesignature matches any specific previous caller. When the voiceauthentication system determines that the voice of the caller matchesone or more previous caller that called on one or more other useraccounts, the voice authentication system may alert another module ofthe online service provider (e.g., a chatbot) or the customer serviceagent accordingly. The voice authentication system may have a set ofsuch models for every important intent that customers call the onlineservice provider frequently or for what reasons fraudsters havehistorically called the online service provider. Once the systemidentifies voice vectors that have called multiple times for differentaccounts, the system may create a voice blacklist of fraudsters that canbe shared.

In an example use case, the voice authentication system may facilitatedetecting malicious activity relating to unverified user accounts. Forexample, a “Know your customer (KYC)” restriction can be placed on auser account when malicious activity associated with the user accounthas been detected (e.g., a fraudulent user has called to perform atransaction on that user account, etc.), which limits the number ofactivities that can be done on the user account including restrictedwithdrawals of any funds from the user account. Such accounts are oftencreated for fraud purposes and once money is contributed into suchaccounts, customer service calls are made to lift the restrictions toallow transfer of money outside the user account. A single customer canmake multiple calls for multiple accounts. Based on an intention of“lifting the KYC flag,” the system may compare the voice vector with thetrained machine learning-based models and detect that the same callerhas called previously with the same intention for multiple accounts. Thecustomer service agent can prompt the caller for more authenticationinformation and place the caller on hold until more verification can becompleted.

FIG. 1 is a block diagram of a networked system 100 suitable forimplementing the anomaly detection system, the analysis system, thebrowser analysis system, and the voice authentication system describedherein. As shown, system 100 may comprise or implement a plurality ofdevices, servers, and/or software components that operate to performvarious methodologies in accordance with the described implementations.Exemplary devices and servers may include device, stand-alone, andenterprise-class servers, operating an OS such as a MICROSOFT® OS, aUNIX® OS, a LINUX® OS, or another suitable device and/or server-basedOS. It can be appreciated that the devices and/or servers illustrated inFIG. 1 may be deployed in other ways and that the operations performed,and/or the services provided by such devices and/or servers may becombined or separated for a given implementation and may be performed bya greater number or fewer number of devices and/or servers. One or moredevices and/or servers may be operated and/or maintained by the same ordifferent entity

System 100 includes a service provider server 110 and an agent device120 in communication over a network 140. Agent device 120 may beutilized to provide training data, view flagged transactions, andprocess additional transaction data to identify transactions indicatinga prohibited transaction including potential fraud. In this regard, anagent may process and review the data with service provider server 110,where service provider server 110 may generate a machine learning modelbased on iteratively training using the training data, and furtherprocess the transaction data using the model to flag furthertransactions. Additionally, service provider server 110 may be used tooutput narratives for flagged transactions based on feature analysisthat caused the machine learning engine to perform a classification.

Agent device 120 and service provider server 110 may each include one ormore processors, memories, and other appropriate components forexecuting instructions such as program code and/or data stored on one ormore computer readable mediums to implement the various applications,data, and steps described herein. For example, such instructions may bestored in one or more computer readable media such as memories or datastorage devices internal and/or external to various components of system100, and/or accessible over network 140.

Agent device 120 may be implemented as a communication device that mayutilize appropriate hardware and software configured for wired and/orwireless communication with service provider server 110. For example, inone implementation, agent device 120 may be implemented as a personalcomputer (PC), a smart phone, laptop/tablet computer, wristwatch withappropriate computer hardware resources, eyeglasses with appropriatecomputer hardware (e.g. GOOGLE GLASS®), other type of wearable computingdevice, implantable communication devices, and/or other types ofcomputing devices capable of transmitting and/or receiving data, such asan IPAD® from APPLE®. Although only one device is shown, a plurality ofdevices may function similarly and/or be connected to provide thefunctionalities described herein.

Agent device 120 includes an alert review application 122, a reportreview application 124, a database 126, and a network interfacecomponent 128. Alert review application 122 may correspond to executableprocesses, procedures, and/or applications with associated hardware. Inother implementations, agent device 120 may include additional ordifferent modules having specialized hardware and/or software asrequired.

Report review application 124 may correspond to one or more processes toexecute software modules and associated components of agent device 120to provide features, services, and other operations associated withtraining a machine learning, deep learning, or other artificialintelligence (AI) model, as well as using the model for detection ofprohibited transactions in transaction data sets. In this regard, reportreview application 124 may correspond to specialized hardware and/orsoftware utilized by a user of agent device 120 that may be used toprovide training and transaction data, as well as review results of asupervised machine learning engine having a model trained for fraudulentpattern recognition and narrative text output. For example, reportreview application 124 may be used to first provide training data and/orsets of data to service provider server 110 that includes transactiondata sets for transaction processed by a financial entity, such as abank or financial institution, payment service provider, or othertransaction processor.

Service provider server 110 may utilize features within the data sets toclassify the transactions according to one or more classifiers, whichmay flag one or more transactions as potentially prohibited based onlaws, rules, or regulations. The data sets may be annotated, and flaggedtransactions may be displayed through report review application 124. Anagent may identify any false positives in the flagging of transactionsas potentially prohibited, which may be provided back to serviceprovider server 110 for retraining (e.g., iteratively and/orcontinuously training) of the machine learning model. The flaggedtransactions may include a narrative displayable through report reviewapplication 124, such as a textual description of the reason forflagging the transaction(s) by the model. After training, agent device120 may further be used to view the results of the model processingother transaction data sets, such as for other transaction processed byone or more entities.

Agent device 120 may further include database 126 stored on a transitoryand/or non-transitory memory of agent device 120, which may storevarious applications and data and be utilized during execution ofvarious modules of agent device 120. Database 126 may include, forexample, identifiers such as operating system registry entries, cookiesassociated with alert review application 122 and/or other applications112, identifiers associated with hardware of agent device 120, or otherappropriate identifiers, such as identifiers used forpayment/user/device authentication or identification, which may becommunicated as identifying the user/agent device 120 to serviceprovider server 110. Database 126 may further include any transactiondata sets used for training and/or processing with a machine learningmodel generated by service provider server 110.

Agent device 120 includes at least one network interface component 128adapted to communicate with service provider server 110. In variousimplementations, network interface component 128 may include a DSL(e.g., Digital Subscriber Line) modem, a PSTN (Public Switched TelephoneNetwork) modem, an Ethernet device, a broadband device, a satellitedevice and/or various other types of wired and/or wireless networkcommunication devices including microwave, radio frequency, infrared,Bluetooth, and near field communication devices.

Service provider server 110 may be maintained, for example, by an onlineservice provider, which may provide identification of prohibitedtransactions, such as fraudulent transactions, in transaction data setsprocessed by a financial or transaction processing entity (includingservice provider server 110) using a machine learning or other AI model.In this regard, service provider server 110 includes one or moreprocessing applications which may be configured to interact with agentdevice 120 to train and utilize the model for prohibited transactionidentification. In one example, service provider server 110 may beprovided by PAYPAL®, Inc. of San Jose, Calif., USA. However, in otherimplementations, service provider server 110 may be maintained by orinclude another type of service provider.

Service provider server 110 of FIG. 1 includes an event detection server150, a transaction processing server 160, and a network interfacecomponent 170. Event detection server 150, transaction processing server160, and other applications 134 may correspond to executable processes,procedures, and/or applications with associated hardware. In otherimplementations, service provider server 110 may include additional ordifferent modules having specialized hardware and/or software asrequired.

In some embodiments, the event detection server 150 may implement thefunctionalities of the anomaly detection system, the analysis system,the browser analysis system, and the voice authentication systemdescribed herein. The event detection server 150 may also be associatedspecialized hardware of service provider server 110 to provide aframework to train a machine learning model for one or more predictionengines that can detect prohibited transactions, such as fraud, byprobabilistically detecting anomalies in transaction datasets, whichwill be described in more detail in FIGS. 2 and 4 . In one or moreimplementations, the event detection server 150 may provide a frameworkfor an analysis platform for actionable insight into user interactiondata, which will be described in more detail in FIGS. 2 and 3 . In otherimplementations, the event detection server 150 may provide a platformfor automated device data retrieval and analysis, which will bedescribed in more detail in FIGS. 5-7 . In still other implementations,the event detection server 150 may provide a voice vector framework forauthenticating user interactions, which will be described in more detailin FIGS. 8-10 .

In various implementations, service provider server 110 includes atleast one network interface component 170 adapted to communicate agentdevice 120 and/or other entities over network 140. In variousimplementations, network interface component 170 may comprise a DSL(e.g., Digital Subscriber Line) modem, a PSTN (Public Switched TelephoneNetwork) modem, an Ethernet device, a broadband device, a satellitedevice and/or various other types of wired and/or wireless networkcommunication devices including microwave, radio frequency (RF), andinfrared (IR) communication devices.

Network 140 may be implemented as a single network or a combination ofmultiple networks. For example, in various implementations, network 140may include the Internet or one or more intranets, landline networks,wireless networks, and/or other appropriate types of networks. Thus,network 140 may correspond to small scale communication networks, suchas a private or local area network, or a larger scale network, such as awide area network or the Internet, accessible by the various componentsof system 100.

The customer device 130, in various implementations, may be implementedas a communication device using any appropriate combination of hardwareand/or software configured for wired and/or wireless communication overthe network 140. For example, in one implementation, the user device maybe implemented as a personal computer (PC), a smart phone, a smart phonewith additional hardware such as NFC chips, BLE hardware etc., wearabledevices with similar hardware configurations such as a gaming device, aVirtual Reality Headset, or that talk to a smart phone with uniquehardware configurations and running appropriate software, laptopcomputer, and/or other types of computing devices capable oftransmitting and/or receiving data.

The customer device 130 may install and execute a client-side serviceapplication 132 received from the transaction processing server 110 tofacilitate one or more transaction processes (e.g., point-of-saletransactions). The client-side service application 132 may allow a userto send payment transaction requests to the transaction processingserver 110, which includes communication of data or information neededto complete the request, such as funding source information. Thecustomer device 130 may include one or more browser applications thatmay be used, for example, to provide a convenient interface to permit auser to browse information available over network 140. For example, inone implementation, the one or more browser applications may beimplemented as a web browser configured to view information availableover the Internet, such as a user account for online shopping and/ormerchant sites for viewing and purchasing goods and/or services.

The customer device 130, in various implementations, may include otherapplications as may be desired in one or more implementations of thepresent disclosure to provide additional features available to the user.For example, the other applications may include security applicationsfor implementing server-side security features, programmatic clientapplications for interfacing with appropriate APIs over network 140, orother types of applications. The other applications may also includeemail, texting, voice and IM applications that allow a user to send andreceive emails, calls, texts, and other notifications through network140. In various implementations, the other applications may includefinancial applications, such as banking, online payments, moneytransfer, or other applications associated with transaction processingserver 110. The other applications include a software program, such as agraphical user interface (GUI), executable by a processor that isconfigured to interface to a user.

The customer device 130 may further include cache 134 stored to atransitory and/or non-transitory memory of customer device 130, whichmay store various applications and data and be utilized during executionof various modules of customer device 130. Thus, cache 134 may include,for example, identifiers such as operating system registry entries,cookies associated with the one or more browser applications and/or theother applications, identifiers associated with hardware of customerdevice 130, or other appropriate identifiers, such as identifiers usedfor payment/user/device authentication or identification, which may becommunicated as identifying customer device 130 to the service providerserver 110. For example, the cache 134 may store device attributes ofthe customer device 130, including, but not limited to, applicationinformation associated with various applications on the customer device130, version information of the various applications, cookieinformation, a language used on the customer device 130, screenattributes such as resolution, size, etc., and other informationassociated with the customer device 130. In various implementations,account information and/or digital wallet information may be stored tocache 134 for use by the customer device 130.

The cache 134, in one implementation, may include at least one useridentifier, which may be implemented, for example, as operating systemregistry entries, cookies, identifiers associated with hardware of thecustomer device 130 (e.g., a media control access (MAC) address), orvarious other appropriate identifiers. The user identifier may includeone or more attributes related to the user of the customer device 130,such as personal information related to the user (e.g., one or more usernames, passwords, photograph images, biometric IDs, addresses, phonenumbers, social security number, etc.) and banking information and/orfunding sources (e.g., one or more banking institutions, credit cardissuers, user account numbers, security data and information, etc.). Invarious implementations, the user identifier may be passed with a userlogin request to the transaction processing server 110 via the network140, and the user identifier may be used by the service provider server110 to associate the user with a particular user account maintained bythe transaction processing server 160.

Customer device 130 includes at least one network interface component170 adapted to communicate with the service provider server 110 and/orthe transaction processing server 160. In various implementations,network interface component 170 may include a modem, an Ethernet device,a broadband device, a satellite device and/or various other types ofwired and/or wireless network communication devices including microwave,radio frequency, infrared, Bluetooth, and near field communicationdevices.

Even though only one customer device 130 is shown in FIG. 1 , it hasbeen contemplated that one or more user devices (each similar tocustomer device 130) may be communicatively coupled with the serviceprovider server 110 via the network 140 within the networked system 100.

FIG. 2 illustrates a block diagram of the event detection server 150,according to an implementation of the present disclosure. The eventdetection server 150 includes a feature extraction module 210, aclassification module 230, a first pass fraud detection module 240, adata profile module 250, a report generation module 260, and a secondpass fraud detection module 270. In various aspects, the networkinterface component 125 includes API 202. The API 202 is coupled to theevent detection server 150 with bidirectional signal paths to the secondpass fraud detection module 270. The feature extraction module 210includes a text preprocessing module 211, a feature extraction engine212, an e-mail features repository 213, a voice recognition module 214,a text feature extraction engine 215, a transcript features repository216, an audio feature extraction engine 217, a voice vectors repository218, an email training dataset 219 and a voice training dataset 220. Insome embodiments, when the event detection server 150 detects acommunication (e.g., an email, a call, a chat) from a user device to theservice provider server 110, the feature extraction module 210 mayextract features from the communication. The extracted features mayinclude textual data features extracted from a text portion of thecommunication and/or audio data features extracted from an audio portionof the communication.

For example, the feature extraction engine 212 of the feature extractionmodule 210 can generate textual data features with the textual data. Insome examples, the textual data refers to text extracted from electronicdocument items received by the text preprocessing module 211. Thetextual data features can be stored in the e-mail features repository213. In some aspects, the voice recognition module 214 can process theaudio properties of a received audio file (e.g., voicemail) to produce atextual transcript of the received audio file. The voice recognitionmodule 214 may utilize standard speech processing techniques totranslate the audio to text. The text feature extraction engine 215 cangenerate textual data features with the textual transcript associatedwith the received audio file. The textual data features from the textfeature extraction engine 215 can be stored in the transcript featuresrepository 216. The voice recognition module 214 may provide a processedspeech datafile that includes speech properties of the received audiofile. The audio feature extraction engine 217 may generate audio datafeatures with the processed speech datafile. The audio featureextraction engine 217 may generate voice vectors with the audio datafeatures, where each portion (or field) of respective voice vectorscorresponds to a different combination of audio data features. Forexample, each combination of audio data features may represent wordand/or phrases of a specific spoken language. The voice vectors can bestored in the voice vectors repository 218. In some aspects, the featureextraction module 210 can map (or transform) the textual data featuresand the audio data features into respective ones of different featurerepresentations. In other aspects, the preprocessing module 231 may betasked to perform the feature representation mapping in lieu of thefeature extraction module 210.

The classification module 230 includes a preprocessing module 231, aclassifier 232, a fraud class module 233, a voice scoring module 234, aphishing class module 235, a malware class module 236, other classmodule 237 and a noise class module 238. In some embodiments, theclassification module 230 may determine an intent of the communicationbased on the extracted features, and may select a machinelearning-trained classifier to classify the communication in one or moreof communication categories.

The first pass fraud detection module 240 includes a clustering engine242 and fraud type clusters 244, 246, and 248. In some embodiments, thefirst pass fraud detection module 204 may use the clustering engine 242to assign the communication to one of the clusters (e.g., one of thefraud type clusters 244, 246, and 248) of past communications based onthe intent and the categories associated with the communication.Embodiments are discussed with reference to FIG. 3 .

The data profile module 250 includes a controller 252, a retrievalengine 254 and a data profile repository 256. The second pass frauddetection module 270 includes an anomaly detection module 280, afeedback module 272 and a remedial action module 274. In someembodiments, the second pass fraud detection module 270 may derive aunique pattern of activity from the communication and may detect one ormore anomalies in the communication based on the unique pattern ofactivity. For purposes of brevity and efficient explanation of thefunctional aspects associated with the components described in FIG. 2 ,the block diagram of FIG. 2 will be described in reference to theprocesses of FIGS. 3 and 4 .

FIG. 3 is a flowchart of an example process 300 of actionable insightanalysis into user interaction data, according to an implementation ofthe present disclosure. One or more of the steps 302-314 of process 300may be implemented, at least in part, in the form of executable codestored on non-transitory, tangible, machine-readable media that when runby one or more processors may cause the one or more processors toperform one or more of the steps 302-314. Some examples of computingdevices, such as computer system 1100 of FIG. 11 may includenon-transitory, tangible, machine readable media that include executablecode that when run by one or more processors (e.g., processor 1112) maycause the one or more processors to perform the steps of process 300. Asillustrated, the process 300 includes a number of enumerated steps, butaspects of the process 300 may include additional steps before, after,and in between the enumerated steps. In some aspects, one or more of theenumerated steps may be omitted or performed in a different order.

The process 300 begins at step 302, where the event detection server 150receives user interaction data (e.g., e-mails, an audio clip of a phonecall, etc.) associated with an interaction between the customer device130 and a service provider server (e.g., the transaction processingserver 160). In some aspects of receiving the user interaction data, thetext preprocessing module 211 may access, through the API 202, textualdata associated with a first type of interaction from a first datastructure in a data repository communicably coupled to the serviceprovider server 110. In some examples, the text preprocessing module 211may prepare the received textual data for processing by the featureextraction engine 212, such as removing any unnecessary terms and/orcharacters that help improve the extraction performance. In one or moreaspects, the voice recognition module 214 may access, through the API202, audio data associated with a second type of interaction from asecond data structure in the data repository. In some aspects, thefeature extraction module 210 may group the textual data with the audiodata into the user interaction data to combine the first type ofinteraction with the second type of interaction. In this respect, thetextual data and the audio data may correspond to different portions ofthe user interaction data. In some aspects, the data repository may beremote and/or external to the service provider server 110. In otheraspects, the data repository may be internal to the service providerserver 110. For example, the data repository is, or includes at least aportion of, the data profile repository 256.

Next, at step 304, the feature extraction module 210 of the eventdetection server 150 processes the user interaction data by extractingone or more features from the user interaction data. In some aspects,the extracted features includes textual data features and audio datafeatures as described herein. In various aspects, the feature extractionmodule 210 can select one of multiple different feature extractionalgorithms based on a comparison of performance metrics between each ofthe different feature extraction algorithms. In some examples, thedifferent feature extraction algorithms include, among others,bag-of-words, term frequency-inverse document frequency (TF-IDF),Doc2Vec, and Latent Dirichlet Allocation (LDA). As such, the featureextraction module 210 can apply the selected feature extractionalgorithm to the user interaction data.

Subsequently, at step 306, the classifier 232 of the event detectionserver 150 classifies the extracted features into one of many actionableinsight categories with a machine learning-trained classifier. Forexample, a first actionable insight category may correspond to afraudulent activity implemented with the fraud class module 233, asecond actionable insight category may correspond to a phishing activityimplemented with the phishing class module 235, a third actionableinsight category may correspond to a malware activity implemented withthe malware class module 236, a fourth actionable insight category maycorrespond to a noise activity implemented with the noise class module238, and a fifth actionable insight category may correspond to other (ormiscellaneous) activities implemented with the other class module 237.The output of the fraud class module 233 is fed to the first pass frauddetection module 240. The output of the phishing class module 235 mayinclude a notification transmitted to a phishing detection system (notshown), indicating that the user interaction may correspond to aphishing scheme.

The phishing detection system can analyze and report phishing networklinks (e.g., URLs) extracted from phishing emails. The output of themalware class module 236 includes signaling that is sent to a malwaredetection system (not shown). The signaling may be first sent to a virusanalysis process to identify whether the manipulated file has beenpreviously identified as malicious and to acquire an associatedsignature that can be used for blocking. If the virus analysis processdoes not return a result, then the malware detection system can receivecustomer input with malware attachments for download into an isolatedvirtual environment. The customer input with malware attachments may beanalyzed for static (code based) and dynamic (interaction with thesandbox system) to identify the impact and report the findings to theagent device 120. The output of the other class module 237 includessignaling that is sent to the agent device 120.

Some examples of the other category may include customer complaintsand/or emails about mobile applications not working properly, queriesfor information regarding products and/or services associated with theservice provider server 110, or the like. In the other category, theclassifier 232 may identify customer complaints regarding usage ofproducts and/or services associated with the service provider server 110that can serve as feedback for use by the transaction processing server160 to identify any underlying issues in the services and/or products.The output of the noise class module 238 includes signaling that ispurged. Some examples of noise may include any data that does not relateto the service associated with the service provider server 110,advertisements, subscription emails and the like. Purging the noiseactivity can help increase the efficiency and accuracy of the system.For purposes of brevity, the remainder of the steps in the process 300will be discussed in reference to the fraudulent activity category, butcan also apply to other categories.

In some implementations, the classifier 232 determines an intent of theinteraction from the feature representations using the machinelearning-trained classifier. In some aspects, the intent corresponds toone of the different actionable insight categories. In someimplementations, the classification module 230 selects one of themachine learning-trained classifiers implemented by the classifier 232based on a comparison of performance metrics between each of the machinelearning-trained classifiers. In some aspects of determining the intentof the interaction, the classifier 232 determines the intent of theinteraction using the selected one of the machine learning-trainedclassifiers.

In some implementations, the feature extraction module 210 accesses userinteraction datasets associated with respective ones of interactionsbetween different communication devices (including the customer device130) and the service provider server 110. The classifier 232 can thendetermine an intent of each of the interactions from extracted featuresassociated with each of the interactions using the machinelearning-trained classifier. In turn, the classifier 232 can classifyeach of the interactions as a respective category of the actionableinsight categories based at least in part on the intent of thatinteraction.

In an offline operation, the classification module 230 may include oneor more processors adapted to generate multiple machine learning-basednetworks based on the actionable insight categories. In some aspects,the machine learning-based networks correspond to the respectiveactionable insight categories. The one or more processors in theclassification module 230 may be adapted to train each of the machinelearning-based networks with a respective training dataset to formdifferent machine learning-trained classifiers. In some aspects, therespective training dataset facilitates supervised learning by includinglabeled interaction data indicating what information pertains to whichof the actionable insight categories. In some aspects of determining theintent of the interaction using the machine learning-trained classifier,the classification module 230 can select the machine learning-trainedclassifier from the machine learning-trained classifiers.

Next, at step 308, the clustering engine 242 of the first pass frauddetection module 240 generates multiple clusters based on the extractedfeatures using one or more clustering algorithms. For example, theclusters may include cluster 244 (depicted as “fraud type cluster 1”),cluster 246 (depicted as “fraud type cluster 2”), and cluster 248(depicted as “fraud type cluster N”). In some examples, the one or moreclustering algorithms include, among others, K-means, Gaussian mixturemodel (GMM), and hierarchical. In one or more implementations, theclustering engine 242 maps the interaction to a first cluster of thedifferent first clusters based at least in part on the determinedintent. In some aspects, each of the first clusters corresponds to aparticular type of activity in a first actionable insight category ofthe different actionable insight categories. In some examples, the firstactionable insight category corresponds a fraudulent activityclassification. In some examples, the particular type of activity maycorrespond to one of multiple types of fraudulent activity (e.g., fraudtype cluster 1, fraud type cluster 2, fraud type cluster N). In someimplementations, the clustering engine 242 determines whether theinteraction maps to at least one of the first clusters and generates anew cluster to be included in the first clusters when the interaction isdetermined to not map to the at least one of the first clusters. In someaspects, the new cluster corresponds to a new type of event in the firstactionable insight category.

Subsequently, at step 310, the event detection server 150 detects one ormore anomalies in at least one of the clusters through an anomalydetection operation. In some aspects of detecting whether the one ormore anomalies are present, the event detection server 150 may utilizethe second pass fraud detection module 270 to generate second clusterswithin the at least one of the first clusters using one or moreclustering algorithms. In some aspects, each of the second clusterscorresponds to a unique pattern of activity associated with theparticular type of activity in the first actionable insight category. Insome implementations, the second pass fraud detection module 270 mayutilize the cluster engine 286 to generate the second clusters. Thesecond pass fraud detection module 270 may determine whether the uniquepattern of activity includes the one or more anomalies for theparticular type of activity. In some aspects, the second pass frauddetection module 270 determines that the particular type of activityrepresents malicious activity when the unique pattern of activity isdetermined to include the one or more anomalies.

Next, at step 312, the event detection server 150 issues a remedialaction for the interaction by further authenticating the customer device130 based on the one or more anomalies detected in the at least one ofthe clusters. In some aspects, the event detection server 150 issues theremedial action based on the mapping of the interaction to the firstcluster. In some implementations, the event detection server 150 issuesthe remedial action using the remedial action module 274. In otherimplementations, the event detection server 150 issues the remedialaction using the report generation module 260.

Subsequently, at step 314, the event detection server 150 provides,through the API 202 to a second communication device (e.g., the agentdevice 120) associated with the service provider server 110, anindication of the one or more anomalies. In some aspects, the eventdetection server 150 generates an alert notification that identifies themalicious activity using the remedial action module 274. In someaspects, in providing the indication, the remedial action module 274sends, through the API 202 to the agent device 120 over the network 140,the alert notification. In some implementations, the remedial actionmodule 274 conditionally sends the alert notification. For example, theremedial action module 274 may determine whether a threshold number ofanomalies is detected in the first cluster. The remedial action module274 generates a report indicating the one or more anomalies when thethreshold number of anomalies is detected. In an aspect, the detectednumber of anomalies exceeds the threshold number of anomalies.

FIG. 4 is a flowchart of an example process 400 of probabilistic anomalydetection and mediation, according to an implementation of the presentdisclosure. One or more of the steps 402-410 of process 400 may beimplemented, at least in part, in the form of executable code stored onnon-transitory, tangible, machine-readable media that when run by one ormore processors may cause the one or more processors to perform one ormore of the steps 402-410. Some examples of computing devices, such ascomputer system 1100 may include non-transitory, tangible, machinereadable media that include executable code that when run by one or moreprocessors (e.g., processor 1112) may cause the one or more processorsto perform the steps of process 400. As illustrated, the process 400includes a number of enumerated steps, but aspects of the process 400may include additional steps before, after, and in between theenumerated steps. In some aspects, one or more of the enumerated stepsmay be omitted or performed in a different order.

The process 400 begins at step 402, where the feature preprocessingmodule 281 of the anomaly detection module 280 receives device data of auser device (e.g., the customer device 130) during a communicationsession between the user device and a service provider server 110. Forexample, a user, through the user device, may communicate with theservice provider server 110 (e.g., accessing a website associated withthe service provider server 110, initiating an electronic transactionwith the service provider server 110, etc.). Upon detecting thecommunication session between the user device and the service providerserver 110, the feature preprocessing module 281 may obtain device dataassociated with the user device. In some aspects, the device data mayinclude device attributes, that in combination, is unique to a type ofdevice (e.g., a manufacturer, a model, etc.) associated with thecustomer device 130. Thus, the combination of the device attributes maybe used by the feature preprocessing module 281 to verify whether theuser device is of a particular type (e.g., a particular manufacturer, aparticular model, etc.) that the user device purports to be. In anaspect, each device attribute in the different of device attributes ismarked with a different descriptor and a corresponding attribute valuethat are specific in identifying the device type of the customer device130. For example, the different device attributes may include, amongothers, a pixel depth attribute, version information of a particularapplication (e.g., a web browser) of the customer device 130, anidentifier, a list of plug-ins installed on the customer device 130, afont used in the customer device 130, a language used in the customerdevice 130, screen attributes such as a width, a height, a resolution,and other attributes.

In some aspects of receiving the device data, the feature preprocessingmodule 281 may receive the device data of the customer device 130 inresponse to occurrence of an event associated with the customer device130. For example, the event occurrence may include a user interactionbetween the customer device 130 and the service provider server 110,such as a connection attempt to the service provider server 110. In someaspects, the event may identify a device type of the customer device 130(e.g., a particular manufacturer, a particular model, etc.) and a typeof the user interaction (e.g., connection attempt).

In other aspects of receiving the device data, the feature preprocessingmodule 281 may receive a first logs indicating a first number of eventsassociated with user devices (e.g., other customer devices 130 d) thatoccur in a first window of time. In some aspects, each event of thefirst number of events may include first device data corresponding tothe first window of time for a user device of the user devices. Thefeature preprocessing module 281 also may receive second logs indicatinga second number of events associated with the user devices that occur ina second window of time different from the first window of time (e.g.,occurring at a later time). In some aspects, each event of the secondnumber of events may include second device data corresponding to thesecond window of time for a user device of the user devices.

In still other aspects of receiving the device data, the featurepreprocessing module 281 may receive user queries. In some aspects, eachof the user queries may include a user interaction between a useraccount of a service (e.g., electronic payment processing) and theservice provider server 110 associated with the service. The featurepreprocessing module 281 may perform an extraction operation, where thefeature preprocessing module 281 may extract metadata of userinteractions from the user queries. In some aspects, the metadataincludes the device data of the customer device 130.

In some implementations, the feature combination module 282 of theanomaly detection module 280 can select a combination of features fromthe different features. In some aspects, the combination of featuresincludes features having a variance of expected values that exceeds athreshold variance. In some aspects of selecting the combination offeatures, the feature combination module 282 selects multiple deviceattribute combinations from the different of device attributes thatrepresent features of interest. In some aspects, each of the multipledevice attribute combinations corresponds to a different combinationamong a subset of the different device attributes.

Next, at step 404, the anomaly detection module 280 selects a predictionengine to process the device data and generate a vector of likelihoodscores for different device attribute combinations. For example, theanomaly detection module 280 may select a prediction engine fromdifferent prediction engines to process the device data, where theselected prediction engine corresponds to a non-parametric statisticalmodel of different of non-parametric statistical models. The anomalydetection module 280 may generate a vector of likelihood scores fordifferent device attribute combinations with different non-parametricstatistical models. In some aspects, the non-parametric estimationmodule 283 may generate a histogram for each of the different deviceattribute combinations of the particular device type and estimate aprobability density function distribution that best fits the histogram.In some aspects, the histogram includes a two-dimensional relationshipbetween first attribute values of a first device attribute againstsecond attribute values of a second device attribute in a given deviceattribute combination of the device attribute combinations. In otheraspects, the histogram can indicate relationships between N deviceattributes in N-dimensional space in a given device attributecombination of the device attribute combinations, where N is a positiveinteger.

In some implementations, in an offline operation, the anomaly detectionmodule 280 generates a first number of non-parametric statistical modelswith first predetermined thresholds based on a first window of time, andthe anomaly detection module 280 generates a second number ofnon-parametric statistical models with a second predetermined thresholdsbased on a second window of time. In some aspects, the second window oftime is subsequent to the first window of time. In one or more aspects,at least one of the first predetermined thresholds is different from atleast one of the second predetermined thresholds.

In some aspects, the non-parametric statistical models include a machinelearning-based models. In one or more implementations, the anomalydetection module 280 can train the machine learning-based models with atraining dataset that indicates user interactions captured within one ormore predetermined windows of time and a mapping of the userinteractions to predetermined probability density functiondistributions. The training of the machine learning-based models canoccur in an offline mode of the anomaly detection module 280. In someaspects of selecting the prediction engine of the prediction engines,the anomaly detection module 280 processes device attributes from thedevice data with the machine learning-based models. In turn, the anomalydetection module 280 can generate a likelihood prediction for each ofthe device attributes with the different machine learning-based models.

In some aspects, each of the machine learning-based models includes oneor more executable programs and/or models configured to initiallyprocess one or more training data sets having transactions processed byan entity, including service provider server 110. The transactions inthe training datasets 219, 220 may include legitimate transactions andmalicious and/or fraudulent transactions, such as those transactionsprohibited due to money launder laws, rules, and regulations when entityengage in illegal and/or malicious behavior. The training data set mayinclude labeled and/or unlabeled data, which may include classificationsof valid transactions and prohibited transactions (e.g., “no fraud” or“potential or detected fraud,” respectively). These may be labeled by ahuman operator, such as an agent that reviews transactions forprohibited activity such as fraud, and the like for reporting to aregulatory agency, body, or entity. Thus, one or more classifiers may beestablished by the agent or entity processing the data, or may bedetermined based on outlier transactions or transactions having featuresindicating prohibited conduct or behavior. Thus, the classifiers may bebuilt and trained so that classifications may be assigned to particulardata points (e.g., transactions) within each of the training datasets219, 220.

The training datasets 219, 220 may include different features, such as aplatform for the transaction (e.g., mobile, web, etc.), an accountnumber, a transaction identifier (ID), a transaction type (e.g.,payment, gambling, etc.), an encrypted transaction ID, a parenttransaction ID, a created and/or update date, a US dollar equivalentamount (e.g., where credits and sent payments may be in a negativeformat), a local currency amount and/or code, a billing and/or shippingaddress, a funding source and/or backup funding source, a bank accountnumber, a bank hash-based message authentication code (HMAC), a cardnumber and/or hash, a card bun HMAC, a card issuer, a balance and/orimpact on a balance due to the transaction, a transaction status and/oritems within the transaction, notes and/or subject lines within messagesfor the transaction, an automated clearinghouse return codes, an ID onanother marketplace or platform, a counterparty name, a counterpartyaccount number, a counterparty account type, a counterparty countrycode, a counterparty email, a counterparty transaction ID, acounterparty ID on a marketplace or platform, a counterparty accountstatus, a referring URL, an IP address, whether the transaction wassuccessful, and a date (e.g., month/year) of transaction.

Other exemplary features and/or categories of features in the trainingdatasets 219, 220 that may be important to training the values andweights of a machine learning model may include risk rules regardingflagging of transactions as incorrect descriptions or messages,complaints and flags by other parties within transactions, gamblingactivities including fantasy sports, specific country accounts andtransaction activities from countries marked as high risk for fraud, asame or similar account owner for a sender and receiver in atransaction, counterfeit flagged accounts, volume of payments in a highrisk transaction corridor or category, a spike in activity ortransaction value after a dormant or inactive period, a number oftransactions and total amount (including if the transactions werecross-border transactions), a previous account takeover flag, amalicious seller flag, an account restriction due to previous malicioususe or rule violation, a cross-border payment from a device usingin-person payment instrument processing (e.g., through processing apayment card EMV chip or magnetic stripe to provide the payment), acheck deposit amount and transfer of deposited funds, a deposit andwithdrawal/transfer of all or a substantial portion of the depositwithin a time period, a gift card usage and withdrawal/transfer of suchfunds, a premier account usage and activity/inactivity, and/or a numberof transactions between the same parties.

When generating machine learning engine 132, the features in thetraining datasets 219, 220 may be used to generate different layers of amachine learning model used to detect the prohibited transactions, whichmay include different nodes, values, weights, and the like. The machinelearning-based model may utilize a supervised machine learningalgorithm, function, or technique that utilizes continuous and/oriterative learning to generate the model. In some implementations, themachine learning-based model may be implemented as a deep learningnetwork, including a convolution neural network, a recurrent neuralnetwork, or a deep neural network. When training the model, the anomalydetection module 280 may utilize feedback and annotations or labelingfrom the agent device 120 to iteratively train the model. For example,transactions in the training data set and/or other data sets may beflagged using the machine learning technique to identify prohibitedtransactions, where the agent device 120 may send an indication that theflagged transactions were not actually prohibited (e.g., not indicativeor including fraud). Identification of these false positives may be usedto retrain the machine learning-based model in a continuous and/oriterative process so that false positives may be reduced and/oreliminated, and the machine learning-based model may more accuratelypredict and detect fraud or other prohibited transactions. Thus, themachine learning-based model is trained for detection of prohibitedtransactions, as well as review of results from the machinelearning-based model that has been modeled for prohibited transactiondetection.

Although the above discussions pertain to an artificial neural networkas an example of machine learning, it is understood that other types ofmachine learning methods may also be suitable to implement the variousaspects of the present disclosure. For example, support vector machines(SVMs) may be used to implement machine learning. SVMs are a set ofrelated supervised learning methods used for classification andregression. A SVM training algorithm—which may be a non-probabilisticbinary linear classifier—may build a model that predicts whether a newexample falls into one category or another. As another example, Bayesiannetworks may be used to implement machine learning. A Bayesian networkis an acyclic probabilistic graphical model that represents a set ofrandom variables and their conditional independence with a directedacyclic graph (DAG). The Bayesian network could present theprobabilistic relationship between one variable and another variable.Other types of machine learning algorithms are not discussed in detailherein for reasons of simplicity.

Subsequently, at step 406, the anomaly detection module 280 determineswhether at least one likelihood score in the vector of likelihood scoresfails to satisfy one or more predetermined thresholds. For example, theanomaly detection module 280 may use the threshold comparator module 284to compare each of the vector of likelihood scores against the one ormore predetermined thresholds to determine whether at least onelikelihood score fails the one or more predetermined thresholds. Theanomaly detection module 280 may also determine that the at least onelikelihood score corresponds to a particular device attributecombination.

Next, at step 408, the anomaly detection module 280 detects an anomalyin the user interaction based on the at least one likelihood scorefailing to satisfy the one or more predetermined thresholds. Forexample, the anomaly detection module 280 may select an anomalydetection engine from different anomaly detection engines to detect ananomaly in the user interaction based on a result of the comparisonoperation that indicates at least one likelihood score in the vector oflikelihood scores does not satisfy the one or more predeterminedthresholds. In some embodiments, the probability density function module285 may process device attributes from the device data with aprobability density function algorithm. The probability density functionmodule 285 can generate a probability density distribution with theprobability density function algorithm. In some aspects, the probabilitydensity distribution includes a two-dimensional relationship betweendifferent likelihood values against different device attribute valuesfor a given device attribute combination from the different deviceattribute combinations. For example, for each device attributecombination in the different device attribute combinations, the anomalydetection engine can generate a likelihood metric indicating that acorresponding attribute value associated with the user interaction is anactual occurrence. In other aspects, the probability density functionmodule 285 may obtain a user profile that indicates a user behaviorassociated with one or more of the user device or a user account of aservice, and the probability density function module 285 may determine alikelihood metric indicating that a corresponding attribute valueassociated with the user interaction is an actual occurrence, based onthe user profile.

In other implementations, the cluster engine 286 of the anomalydetection module 280 can apply a clustering algorithm to a vector oflikelihood scores to generate one or more clusters. This may be analternative track to the use of the non-parametric estimation module 283and the probability density function module 285. The outlier detectionmodule 289 of the anomaly detection module 280 can detect a data pointcorresponding to at least one likelihood score of the vector oflikelihood scores that is outside of the one or more clusters. Theoutlier detection module 289 can apply a Euclidean multi-dimensionaldistance calculation to determine a distance from the one or moreclusters in N-dimensional space, where N is a positive integer. Theoutlier detection module 289 determines whether the data point is amember of the one or more clusters or an outlier based on the distance.In some aspects, the outlier detection module 289 determines that thedata point is an outlier based on the distance being greater than athreshold distance. In this respect, the outlier corresponds to theanomaly in the user interaction.

In still other implementations, the singular value decomposition engine287 of the anomaly detection module 280 can apply a singular valuedecomposition algorithm to a vector of likelihood scores to determineone or more principle components in the vector of likelihood scores interms of eigenvalues and eigenvectors. In some aspects, each of the oneor more principle components identifies which of the device attributeshas a highest variance based at least on a ranking of the eigenvectorsby order of variance. The principle component analysis engine 288 of theanomaly detection module 280 then applies a projection operation bymatrix multiplication to project new attribute values in a direction ofa principle component through an eigenvector that corresponds to theprinciple component. In turn, the principle component analysis engine288 can generate a cluster in an eigenspace with the projected newattribute values. The outlier detection module 289 can then detect adata point corresponding to at least one projected new attribute valuethat is outside of the cluster. The outlier detection module 289 appliesa Euclidean multi-dimensional distance calculation to determine adistance from the cluster and determines whether the data point is amember of the cluster or an outlier based on the distance. In someaspects, the outlier detection module 289 determines that the data pointis an outlier based on the distance being greater than a thresholddistance. In this respect, the outlier corresponds to the anomaly in theuser interaction.

Subsequently, at step 410, the remedial action module 274 of the secondpass fraud detection module 270 issues a remedial action to the customerdevice 130 through the API 202 in response to detecting the anomaly inthe user interaction. For example, the remedial action module 274 mayfurther authenticate the customer device 130 when the anomaly in theuser interaction is detected.

In some aspects, in issuing the remedial action, the remedial actionmodule 274 issues, through the API 202, a first remedial action thatdenies the customer device 130 with access to the service when the atleast one likelihood score is lesser than a first predeterminedthreshold. In some aspects, the remedial action module 274 also mayissue a second remedial action that prompts the customer device 130 toprovide additional authentication information when the at least onelikelihood score is greater than the first predetermined threshold andlesser than a second predetermined threshold. In other aspects, theevent detection server 150 may grant the customer device 130 with accessto the service when each likelihood score in the vector of likelihoodscores is greater than the second predetermined threshold.

In some implementations, the remedial action module 274 sends one ormore messages to the customer device 130 as a next remedial action toprompt the customer device 130 to supply further authenticationinformation, when an anomaly in the user interaction is detected. Inturn, the feedback module 272 of the second pass fraud detection module280 may receive user feedback in response to the one or more messages.The feedback module 272 may aggregate the feedback and/or post-processthe user feedback to a suitable format. The feedback module 272 canprovide the user feedback to the non-parametric statistical models inthe non-parametric estimation module 283. In some aspects, the userfeedback indicates updated authentication information of the customerdevice 130. In turn, the anomaly detection module 280 may adjust the oneor more predetermined thresholds based on the user feedback.

FIG. 5 illustrates a block diagram of a data profile module 250 forbrowser data retrieval and analysis, according to an implementation ofthe present disclosure. The data profile module 250 represents a novelframework for in-depth and automatic testing of current and pre-releaseweb browser applications with the intent of generating a comprehensivedatabase of their behaviors, features, and functionality in order toprovide additional capabilities for detection and mitigation ofmalicious actors using modified or spoofed client devices. The dataprofile module 250 may allow baselining for statistical modeling ofbrowser traffic and can generate alert notifications of newfunctionality that can be implemented into anti-fraud defensemechanisms.

The data profile module 250 includes a retrieval engine 254 that isadapted to fetch device data, such as web browser information frommultiple data sources 510. The data sources 510 may include browserrepositories 510 a-510 c storing information associated with differenttypes of browsers (e.g., different manufacturers, different models,different versions which include current and also pre-release versionsand models, etc.) and a fraud tool repository 510 d storing informationassociated with known tools used by malicious users for launchingattacks on web servers. The data profile module 250, knowledgeable aboutdifferent vendor's browser release channels and their typical downloadURLs, regularly fetches new versions. This may be in response to a newrelease, or may be part of a regular daily process. These may bepre-compiled binary downloads, or the data profile module 250 itself maybe able to compile instances of the browser from retrieved source code.Thus, the data profile module 250 may retrieve or otherwise obtainexecutable code associated with different browser applications (e.g.,from different manufacturer, different models, different versions,etc.). For example, the data profile module 250 may obtain a GoogleChrome browser version 2.1, a Google Chrome browser version 3.0, anApple Safari browser version 11.0, an Apple Safari browser version 12.0,a Microsoft Edge browser version 3.2, etc., and store the differentbrowser applications in the data profile repository 256.

Subsequently, the controller 252 of the data profile module 250 candeploy a browser instance 530 (associated with any one of the browserapplications stored in the data profile repository 256) in a testingenvironment 502. In some embodiments, the testing environment 502 mayinclude a virtual machine or a container using virtualizationtechnologies. For example, the controller 252 can deploy the browserinstance 530 in a virtual machine. The intent is to have an entirely‘clean’ instance of the browser with no customizations or user data thatmay change its behavior from the pre-set defaults. This automationprocess to deploy the browser instance 530 can target several differentmethods, for example: (1) deployment of the browser instance 530 to anew virtual machine, using a fully virtualized environment, (2)deployment of the web browser instance 530 to a segregated container,(3) automation of the web browser instance 530 on a hardware device, forexample, using a remote automation process to remotely control the webbrowser on a device (e.g., a smart phone such as an ANDROID™ phone), and(4) automation of the web browser instance 530 on an emulated device,for example, using the web browser instance 530 (e.g., SAFARI™, etc.)within an emulator.

In some implementations, the browser instance 530 of the fetched webbrowser is invoked in the testing environment 502. Using a browserautomation process, the browser instance 530 can control and run througha series of interactions with a web server 540. For example, the browserinstance 530 can interact with the web server 540 via the virtualmachine. The controller 252 can inspect the data that the web browserinstance 530 sends for anti-fraud purposes, and can run programming code(e.g., JavaScript fingerprinting code, etc.) in an attempt to identifyusers over time for anti-fraud purposes. Most importantly, the dataprofile module 250 can detect spoofed web browsers running on thecustomer device 130 used for abuse and financial fraud based oncomparing the device data obtained from the web browsers running on thecustomer device 130 and the data obtained through the automated analysisprocess via the interactions between the browser instance 530 and theweb server 540 as described herein. These features may includenetwork/communication details, such as: (1) The HTTP headers the browsersends, in a brand new unmodified install, (i) the values that areprovided, (ii) the order the headers are sent in, and (iii) the headersthat are sent in what conditions; (2) when connecting to a secure site,the TLS ciphers that the web browser supports, and in what preferenceorder they are requested in; and (3) the protocols the web browsersupports (e.g., HTTP 1.0, 1.1, QUIC, 2.0, 3.0, etc.) and the supportedcommunication methods (e.g., Websockets, WebRTC, etc.)

The programming code (e.g., the JavaScript fingerprinting code) that isused for fingerprinting can identify device features, such as: (1) thedocument object model (DOM) processes and attributes the web browserclaims to support; (2) the order that the web browser lists theseattributes, when requested; (3) an indication of whether theaforementioned processes appear to be “native” values generated by thebrowser software, or overwritten by the user; (4) the browser-specificprocesses and attributes it exposes (e.g. FIREFOX™ exposesFIREFOX™-specific CSS attributes with the “-moz-” prefix); (5) how theweb browser handles and responds to error conditions, including thosedeliberately invoked (e.g., when executing JavaScript code to evaluatethe mathematical expression “ThisIsNotANumber/0”, CHROME™ returns theerror message “Uncaught ReferenceError: ThisIsNotANumber is not defined”whereas SAFARI™ returns the error message “ReferenceError: Can't findvariable ThisIsNotANumber”).

By controlling the browser's actions and having the ability to servecontent to it and record its responses and behaviors, the data profilemodule 250 can thus record both client-side and server-side details. Forexample, a remote process can be invoked to serve as an automationtooling mechanism to drive the web browser instance 530 through a suiteof tests and actions. The browser instance 530 can record the webbrowser traffic to the web server 540 as well as the return signalingfrom the web server 540. The web server 540 can record its outgoing webbrowser traffic to the browser instance 530. The recorded client-sidedata and server-side data can be stored in a data structure to maintaina database of web browser behavior, attributes, functionality, etc.

There are several use cases that the data profile module 250 can beapplicable. For example, the data profile module 250 can be used foralerting of new or forthcoming attributes or behaviors in web browsers.For example, a currently-released version of a particular web browsermay expose a particular number of JavaScript DOM attributes (e.g., 100),as determined through the automated deployment process described herein.Through automation, the data profile module 250 can download the latestBeta version and observe that the number of JavaScript DOM attributesexposed by the latest Beta version is different from the current version(e.g., 103 attributes). The data profile module 250 can alert adeveloper or analyst, giving them the name and location of theadditional attribute(s) that is not exposed by the current version, itsdefault value, and how it behaves in certain forced-error conditions.The data profile module 250 may also determine if the additionalattribute(s) is useful for anti-fraud purposes, and implement theadditional attribute(s) within a fingerprinting mechanism such that whenthe new version of the particular web browser is released, the dataprofile module 250 can be updated with new defensive capabilities.

In another use case, the data profile module 250 can be used foralerting of changed or removed attributes or behaviors. Similarly to thefirst use case, if the data profile module 250 determines that a usefulfeature used by an anti-fraud system is removed in an upcoming version,or that it now returns a different result or behaves in a different waythan previously observed, the data profile module 250 can determinemitigations or alternative approaches (e.g., updating the frauddetermination process to reflect the changes) instead of taking areactive action when data quality suddenly drops or fraud increases.

In another use case, the data profile module 250 baseline the features.The data profile module 250 may perform risk scoring based on technicaldata that the web browser sends to an online service provider (e.g.,service provider server 110). For example, the technical data mayindicate that the web browser supports a specified number of encryptionciphers and indicates a prioritized order for the ciphers. This type ofinformation may be part of the browser's internal operations and is nottypically modifiable by the user. The data profile module 250 can overtime observe typical patterns of data associated with a particularbrowser version. The data profile module 250 can determine “known good”baseline profiles that can be used to accelerate this learning process.This known good data can then be sent to an analyst (e.g., to the agentdevice 120) for implementation in a rule or model, can be fedautomatically into a model for use in training, or can be fed into alive risk assessment for immediate use.

In still another use case, the data profile module 250 can be used foranalyzing criminal fraud and/or abuse tools. For example, maliciousattackers may invoke anti-detect processes to submit spoofed browserdata when interacting with the online service provider. The spoofedbrowser data is used to hide the fact that these malicious attackers areusing such a tool, instead pretending to be a normal web browser. Inother examples, the malicious attackers may spoof certain attributes inan attempt to appear like a particular customer who they know uses acertain browser profile. It is common for these types of tools to bebuilt on top of existing web browser processes with additionalmodifications made. The data profile module 250 may load copies ofcriminal fraud tools and profile them in depth, comparing results to alegitimate version of the web browser application in order to determinethe exact changes made by the malicious attacker, which can then be usedto implement anti-fraud defenses against the tool.

In yet another use case, the data profile module 250 can be used fordetection of mismatching features. The data profile module 250 maydetect valid but mismatching feature sets compared to the claimed agentdevice. For example, a client (e.g., the customer device 130) thatclaims to run a non-desktop version of a select web browser but has thefeature set and attributes of a desktop version of another select webbrowser. The data profile module 250 may determine that the client isinvolved in malicious activities with the service provider server 110when such a mis-match is detected.

In still yet another use case, the data profile module 250 can sendprecisely targeted anti-fraud challenges to suspect devices. Forexample, existing anti-fraud defenses may examine particular DOM valuesor execute particular code statements in an attempt to fingerprint theclient device. These defenses are commonly designed to work on as manydevice types and browser versions as possible, to ensure effectivecollection from the widest possible audience. With granular visibilityinto all browser versions, the data profile module 250 may deployhighly-targeted fingerprinting code. For example, if the data profilemodule 250 can determine that a particular version of FIREFOX™ has onehundred unique attributes, the data profile module 250 may prompt eachclient for three different attributes (out of the one hundred uniqueattributes), thus yielding over 20 million potential combinations oftests. This can significantly increase the complexity for an adversaryto gain a complete understanding of the tests that the data profilemodule 250 can run.

FIG. 6 conceptually illustrates an exemplary workflow 600 of theautomated device data retrieval and analysis of FIG. 5 , according to animplementation of the present disclosure. For purposes of brevity andefficient explanation of the functional aspects associated with thecomponents described in FIG. 6 , the block diagram of FIG. 6 will bedescribed in reference to the processes of FIG. 7 .

FIG. 7 is a flowchart of an example process 700 of automated device dataretrieval and analysis, according to an implementation of the presentdisclosure. One or more of the steps 702-714 of process 700 may beimplemented, at least in part, in the form of executable code stored onnon-transitory, tangible, machine-readable media that when run by one ormore processors may cause the one or more processors to perform one ormore of the steps 702-714. Some examples of computing devices, such ascomputer system 1100 may include non-transitory, tangible, machinereadable media that include executable code that when run by one or moreprocessors (e.g., processor 1112) may cause the one or more processorsto perform the steps of process 700. As illustrated, the process 700includes a number of enumerated steps, but aspects of the process 700may include additional steps before, after, and in between theenumerated steps. In some aspects, one or more of the enumerated stepsmay be omitted or performed in a different order.

The process 700 begins at step 702, where the retrieval engine 254 ofthe data profile module 250 receives, over the network 140, device dataassociated with an application over a communication channel from one ormore data sources. In some aspects, the retrieval engine 254 may receivedevice data associated with multiple applications (e.g., differentmanufacturers, different models, different versions, etc.) from the oneor more data sources. For example, the retrieval engine 254 can receivefirst device data that corresponds to a first version of differentversions for the at least one application, for example, from one of thebrowser repositories 510 a-c. In other aspects, the retrieval engine 254can receive the first device data and second device data thatcorresponds to a second version of the different versions that at leastin part different from the first version, for example, from another oneof the browser repositories 510 a-c. In various implementations, theretrieval engine 254 is communicably coupled to the API 202.

Next, at step 704, the controller 252 of the data profile module 250invokes an instance (e.g., the browser instance 530) of the applicationin a remote processing environment (e.g., the testing environment 502)using the device data. The remote processing environment is remote tothe customer device 130. For example, at action 610, the instance of aweb browser is invoked and controlled by an automation processexecutable in the remote processing environment. In some aspects ofinvoking the instance of the at least one application in the remoteprocessing environment, the controller 252 deploys the at least oneapplication to a virtual machine of different virtual machines using avirtualized environment. In other aspects of invoking the instance ofthe at least one application in the remote processing environment, thecontroller 252 deploys the at least one application to a segregatedcontainer accessible to the controller of the service provider server110. In still other aspects of invoking the instance of the at least oneapplication in the remote processing environment, the controller 252establishes a connection with the instance of the at least oneapplication executed on a remote communication device. In this respect,the controller 252 can establish remote control of the instance of theat least one application through the connection to the remotecommunication device. In yet other aspects of invoking the instance ofthe at least one application in the remote processing environment, thecontroller 252 can establish a connection with the instance of the atleast one application executed on a remote emulation device. In thisrespect, the controller 252 can establish remote control of the instanceof the at least one application through the connection to the remoteemulation device.

Subsequently, at step 706, the controller 252 causes the instance of theapplication to interact with a web server over the network. For example,the controller 252 can send, to a remote processing environment, acontrol message that causes the instance of the at least one applicationto interact with the web server 540 over the network 140. In one or moreimplementations, the controller 252 sends, to the remote processingenvironment, a first control message that prompts the instance of the atleast one application to send a request to the web server 540 forreturning a process script that is executable on the remote processingenvironment. For example, at action 620, the first control message mayinclude an instruction to the web browser to request “Test 1.”Thereafter, at action 622, the instance of the web browser transmits arequest for “Test 1.” At action 624, the web server 540 transmitsinterrogation scripts and web page documents to the web browser inresponse to the request. For example, at action 626, the web server 540responds to the request by transmitting a return message containingscript code for executing “Test 1.”

Next, at step 708, the controller 252 can access traffic data associatedwith one or more interactions between the instance of the applicationand the web server. In some aspects, the traffic data indicates abehavior of application data associated with the at least oneapplication based on the one or more interactions with the web server540. In some aspects of the accessing the traffic data, the controller252 can acquire first traffic data transmitted from the instance of theat least one application to the web server. In some aspects, the firsttraffic data includes one or more of (1) behavioral patterns associatedwith the at least one application in the one or more interactions over apredetermined period of time, (2) multiple attributes of the at leastone application that is set for the one or more interactions, or (3)multiple functionality parameters of the at least one application thatis activated in the one or more interactions.

In some aspects of accessing the traffic data, the controller 252executes the process script on the remote processing environment withthe instance 530 of the at least one application. The controller 252 caninterrogate the at least one application with the executed processscript. In some aspects, the interrogating includes the one or moreinteractions between the instance 530 and the web server 540. Thecontroller 252 can then log results of the interrogating in a datastructure of the data profile repository 256. For example, at action632, the controller 252 can record client-side activity data (e.g.,occurring at the browser instance 530), such as DOM attributes andvalues and/or generated error messages, among others. At action 634, thecontroller 252 can record data sent from the browser instance 530 to theweb server 540, such as supported TLS ciphers, HTTP request headers,among others. In some aspects of interrogating the at least oneapplication with the executed process script, the controller 252 caninvoke a predetermined error condition in the one or more interactions.As such, the controller 252 can log a response of the browser instance530 of the at least one application based on the predetermined errorcondition.

Subsequently, at step 710, the controller 252 determines one or morefeatures of the application in a native state from the traffic data. Asdiscussed above, the traffic data may indicate the behavior of theapplication. Thus, the controller 252 may determine the one or morefeatures of the application based on the behavior of the application. Insome examples, values in the native state may correspond to defaultvalues (or reset values) set for attributes of the at least oneapplication.

Next, at step 712, the controller 252 generates a data profile of theapplication that indicates the one or more features in the native state.In some aspects of generating the data profile, the controller 252 cangenerate a data structure that includes application data thatcorresponds to the first version. In some aspects, the application dataindicates one or more of (1) behavioral patterns associated with thefirst version over a predetermined period of time, (2) multipleattributes of the at least one application that correspond to the firstversion, or (3) multiple functionality parameters of the at least oneapplication that correspond to the first version. In some aspects ofgenerating the data profile, the controller 252 can generate a firstdata structure that comprises first application data the corresponds tothe first version and a second data structure that comprises secondapplication data that corresponds to the second version. In someaspects, the first data structure and the second data structure areprioritized based on a first weighting associated with the firstapplication data and a second weighting associated with the secondapplication data. In some aspects, the first weighting and the secondweighting are based at least in part on respective lifecycles of thefirst application data and the second application data.

Subsequently, at step 714, malicious activity performed by a user devicemay be detected based on the data profile. For example, the controller252 provides the traffic data and the data profile to the data profilerepository 256 accessible to one or more remote devices associated witha service. The remote devices may detect malicious activity performed bythe customer device 130 by comparing device data associated with thecustomer device 130 and the data profile. For example, when the customerdevice 130 indicates that the application that is in communication withthe service provider server 110 is of a particular type (e.g., aparticular manufacturer, a particular model, a particular version,etc.), the remote device may compare the device data received from thecustomer device 130 (which may include data associated with interactionsbetween the customer device 130 and the service provider server 110)against the data profile associated with an application of theparticular type. When the device data deviates from the data profile bymore than a threshold, the remote device may determine that the customerdevice 130 is used to conduct malicious activity (e.g., accounttake-over, etc.). In some implementations, the controller 252 generatesa notification that indicates the behavior of the application dataassociated with the at least one application. The controller 252 mayprovide, through the API 202 over the network 140, the notification tothe customer device 130. In various aspects, the controller 252 providesthe data profile from the data profile repository 256 to a remote engineassociated with the service that performs one or more detectionoperations with the data profile and detects potential maliciousactivity associated with the at least one application from the one ormore detection operations. For example, at action 640, the stored datacan be used to enhance anti-fraud capabilities.

FIG. 8 conceptually illustrates an exemplary workflow 800 of a voicevector framework for authenticating user interactions, according to animplementation of the present disclosure. For purposes of brevity andefficient explanation of the functional aspects associated with thecomponents described in FIG. 8 , actions of the workflow diagram of FIG.8 will be described in reference to the processes of FIGS. 9 and 10 . Ingeneral, FIG. 8 illustrates actions that authenticate a caller who callsthe service provider server 110 in association with a user account withthe service provider server 110. For example, the caller may be callingto perform a transaction in association with the user account (e.g.,resetting a password, performing a payment transaction, etc.). Theworkflow 800 may include actions for authenticating the caller foraccessing the user account (or for performing the transaction for theuser account) based on voice data associated with the caller.

In some embodiments, according to the workflow 800, the voice vectorframework first authenticates the caller as legitimate customer and thenchecks if the same caller has called before with similar intentions onmultiple accounts. The framework can provide real-time feedback byalerting customer service agents of such suspicious activity. Theframework can leverage various aspects of speech recognition and voiceidentification technology as well as intent identification on theincoming customer call. The framework includes a process of generating asuite of machine learning-based models that encompasses the variationsthat are representative an entity's customers. The framework provides atwo-stage procedure that first verifies the identity of a caller andsecond to check if the same caller has previously called with the sameintention on a different account not belonging to him or her. Thisframework can identify fraudsters, generate a voice blacklist, and alertcustomer service agents in real time to mitigate any security lapse.

FIG. 9 is a flowchart of an example process 900 of a voice vectorframework for verifying user interactions, according to animplementation of the present disclosure. One or more of the steps902-916 of process 900 may be implemented, at least in part, in the formof executable code stored on non-transitory, tangible, machine-readablemedia that when run by one or more processors may cause the one or moreprocessors to perform one or more of the steps 902-916. Some examples ofcomputing devices, such as computer system 1100 may includenon-transitory, tangible, machine readable media that include executablecode that when run by one or more processors (e.g., processor 1112) maycause the one or more processors to perform the steps of process 900. Asillustrated, the process 900 includes a number of enumerated steps, butaspects of the process 900 may include additional steps before, after,and in between the enumerated steps. In some aspects, one or more of theenumerated steps may be omitted or performed in a different order.

The process 900 begins at step 902, where the feature extraction module210 receives voice data associated with a voice communication between auser of a user device and a service provider server. For example, when acaller uses a device (e.g., the customer device 130) to make a phonecall to an agent (e.g., a human agent, a robot, etc.) of the serviceprovider server 110, the voice vector framework, at action 810, mayreceive the voice data associated with the phone call.

Next, at step 904, the feature extraction module 210 extracts userattributes from the voice data. For example, the feature extractionmodule 210 can extract words and/or phrases from the voice transcript.

Subsequently, at step 906, the controller 252 obtains user accountinformation from a user account associated with the voice communication.For example, based on the extracted user attributes, the controller 252may determine that the voice communication is regarding a user accountwith the service provider server 110 (e.g., disputing a transactionassociated with the user account, resetting the user account, etc.).

Next, at step 908, the classification module 230 can select a classifierthat corresponds to a select combination of features based on the useraccount information. For example, at action 822, the classificationmodule 230 can retrieve a trained model based on the account informationof the caller generated by the feature extraction module 210. In someexamples, the trained model may be a gender-age-accent model. In someaspects of selecting the classifier that corresponds to thepredetermined combination of features, the classification module 230 canselect a select number of classifiers that correspond to different userattribute combinations based on different attributes in one or more ofthe user attributes or the user account information.

Subsequently, at step 910, the classification module 230 applies theclassifier to the user attributes and/or the voice vectors.

Next, at step 912, the audio feature extraction engine 217, incoordination with the classifier 232, can generate a voice vectorcomprising multiple scores based on the applied classifier. For example,at action 820, the audio feature extraction engine 217 can generate thevoice vector with a representation of words and/or phrases as audioattributes extracted from the audio call. In some aspects, each of thescores in the voice vector indicates a likelihood that an attribute ofthe different user attributes corresponds to an attribute of the selectcombination of features. In some aspects of generating the voice vector,the audio feature extraction engine 217 can mask one or more of thescores based at least in part on one or more attributes in the useraccount information. In this regard, the audio feature extraction engine217, in coordination with the classifier 232, can generate the voicevector with the masked one or more of the scores. In some aspects, themasked one or more of the scores are excluded from the comparingoperation in step 914. In other aspects of generating the voice vector,the audio feature extraction engine 217 can apply a weighting to one ormore of the scores based at least in part on one or more attributes inthe user account information. In this regard, the audio featureextraction engine 217, in coordination with the classifier 232, cangenerate the voice vector with the weighting applied to the one or moreof the scores.

Subsequently, at step 914, the voice scoring module 234 may compare thevoice vector to a baseline vector that corresponds to a predeterminedcombination of features (e.g., the selected combination of features). Insome aspects of comparing the voice vector to the baseline vector, thevoice scoring module 234 can determine a distance between one or more ofthe scores in the voice vector and corresponding values in the baselinevector. In this respect, the voice scoring module 234 can determinewhether the distance between each of the one or more of the scores andthe corresponding values exceeds a predetermined distance threshold. Insome aspects of comparing the voice vector to the baseline vector, thevoice scoring module 234 can determine whether a difference between thevoice vector and the baseline vector is within a predetermined tolerancethreshold. In this regard, the voice scoring module 234 can determinethat the user device is verified based on the difference between thevoice vector and the baseline vector is determined to be within thepredetermined tolerance threshold. In other aspects of comparing thevoice vector to the baseline vector, the voice scoring module 234 canobtain multiple voice vectors from user account information associatedwith respective ones of multiple different user accounts, and the voicescoring module 234 can compare the voice vector to each of the voicevectors.

Next, at step 916, the report generation module 260 may send anotification to the agent device 120 associated with the serviceprovider server 110. For example, at action 824, the report generationmodule 260 can provide an indication of the caller verification to theagent device 120. In some aspects, the notification includes anindication of whether the user device is verified based on thecomparing. In some aspects, the report generation module 260 generatesthe notification with the indication that indicates that the user deviceis verified when the voice scoring module 234 determines that thedistance between each of the one or more of the scores and thecorresponding values does not exceed the predetermined distancethreshold. In other aspects, the report generation module 260 generatesthe notification with the indication that indicates that the user deviceis not verified when the voice scoring module 234 determines that thedistance between the one or more of the scores and the correspondingvalues exceeds the predetermined distance threshold. In this regard, thenotification prompts the agent device 120 to send one or more queries tothe customer device 130 for additional verification information. Forexample, at action 826, if the caller is verified, notification can besent to the agent device 120. Otherwise, an alert is generated and sentto the agent device 120 for additional verification.

FIG. 10 is a flowchart of an example process 1000 of a voice vectorframework for detecting malicious activity in user interactions,according to an implementation of the present disclosure. One or more ofthe steps 1002-1020 of process 1000 may be implemented, at least inpart, in the form of executable code stored on non-transitory, tangible,machine-readable media that when run by one or more processors may causethe one or more processors to perform one or more of the steps1002-1020. Some examples of computing devices, such as computer system1100 may include non-transitory, tangible, machine readable media thatinclude executable code that when run by one or more processors (e.g.,processor 1112) may cause the one or more processors to perform thesteps of process 1000. As illustrated, the process 1000 includes anumber of enumerated steps, but aspects of the process 1000 may includeadditional steps before, after, and in between the enumerated steps. Insome aspects, one or more of the enumerated steps may be omitted orperformed in a different order.

The process 1000 begins at step 1002, where the feature extractionmodule 210 receives voice data associated with a voice communicationbetween a user of a user device and a service provider server. Forexample, the voice communication may be from a phone call from a user ofthe customer device 130 and an agent (e.g., a human agent, a robot)associated with the service provider server 110. The phone call may beregarding a user account (e.g., disputing a transaction associated withthe user account, resetting a password for the user account, etc.).

Next, at step 1004, the feature extraction module 210 can extract userattributes from the voice data. For example, at action 810, the featureextraction module 210 can extract words and/or phrases from the voicetranscript.

Subsequently, at step 1006, the classification module 230 determines anintent of the voice communication based on the user attributes. Forexample, at action 832, the classification module 230 may determine thatthe intent of the voice call is a request to reset a password for a useraccount. Other example intents may include account verification, liftrestriction from a user account, among others.

Next, at step 1008, the classification module 230 may select one or moreof several machine learning-trained classifiers that correspond to thedetermined intent. As discussed, various voice models corresponding todifferent intents have been trained using historic voice data associatedwith one or more user accounts. After a call has been classified as oneof the different intents, the voice data associated with the call may beused to train the voice model corresponding to the classified intent. Ataction 836, the classification module 230 retrieves trained models basedon the intent. (inventors: can you elaborate on how the In someimplementations, the classification module 230 may obtain user accountinformation from a user account associated with the customer device 130,and the classification module 230 selects a machine learning-trainedclassifier that corresponds to a select combination of features based onthe user account information.

Next, at step 1010, the classifier 232 applies each of the one or moremachine learning-trained classifiers to the user attributes. In someaspects, the classifier 232 applies a selected machine learning-trainedclassifier to the different user attributes.

Subsequently, at step 1012, the audio feature extraction engine 217, incoordination with the classifier 232, generates voice vectors for theone or more machine learning-trained classifiers. For example, the audiofeature extraction engine 217 may generate the voice vectors based on abaseline voice vector associated with each of the one or more of themachine learning-trained classifiers. In some aspects, the audio featureextraction engine 217, in coordination with the classifier 232,generates a voice vector that includes scores based on the appliedmachine learning-trained classifier. In some aspects, each of the scoresin the voice vector indicates a likelihood that an attribute of thedifferent user attributes corresponds to an attribute of a predeterminedcombination of features.

Next, at step 1014, the classifier 232 can select a machinelearning-trained classifier that corresponds to a voice vector havingthe highest aggregate score. For example, the classifier 232 may comparethe scores associated with the different voice vectors of the differentmachine learning-trained classifiers. The score may represent asimilarity between the voice vector and the user attributes extractedfrom the voice data of the voice communication. Thus, the higher thescore, the more similar the voice in the voice communication is to thevoice vector. For example, at action 838, the classifier 232 identifieswhich voice the caller matches in identifying a matched model.

Subsequently, at step 1016, the fraud class module 233, in coordinationwith the voice scoring module 234, may determine a voice signatureassociated with the selected one of the machine learning-trainedclassifiers.

Next, at step 1018, the fraud class module 233 can determine whether thecustomer device 130 interacted with the service provider server 110 in aprior interaction associated with a different user account based on thevoice signature. For example, at action 840, the fraud class module 233can retrieve account information for which the same caller called forreview in a prior occasion. If a match is found, the fraud class module233, in coordination with the report generation module 260, can alert anagent associated with the agent device 120, indicating that the currentcaller called previously for a different user account. Otherwise, thefraud class module 233 may not generate nor send a notification to theagent device 120. In some aspects of determining whether the customerdevice 130 interacted with the service provider server 110, the fraudclass module 233 can access user account information in a datarepository communicably coupled to the service provider server 110 andcompare the voice signature to a historical voice signature associatedwith the accessed user account information that is stored in a datastructure of the data repository. The fraud class module 233 candetermine that a distance between the voice signature and the historicalvoice signature is within a predetermined tolerance threshold. In thisregard, the fraud class module 233 determines that the customer device130 interacted with the service provider server 110 in the priorinteraction. In some aspects of comparing the voice signature to thehistorical voice signature, the fraud class module 233 can obtaindifferent historical voice signatures from user account informationassociated with respective ones of different user accounts, and thefraud class module 233 compares the voice signature to each of thehistorical voice signatures.

Subsequently, at step 1020, the report generation module 260 can send anotification to a communication device associated with the serviceprovider server 110 (e.g., the agent device 120). In some aspects, thenotification may include an indication of whether the user deviceinteracted with the service provider server in the prior interaction.For example, at action 834, the report generation module 260 sendsnotification of detected fraudulent callers to the agent device 120.

FIG. 11 is a block diagram of a computer system suitable forimplementing one or more components in FIGS. 1 and 2 , according to animplementation. In various implementations, the communication device maycomprise a personal computing device e.g., smart phone, a computingtablet, a personal computer, laptop, a wearable computing device such asglasses or a watch, Bluetooth device, key FOB, badge, etc.) capable ofcommunicating with the network. The service provider may utilize anetwork computing device (e.g., a network server) capable ofcommunicating with the network. It should be appreciated that each ofthe devices utilized by users and service providers may be implementedas computer system 1100 in a manner as follows.

Computer system 1100 includes a bus 1102 or other communicationmechanism for communicating information data, signals, and informationbetween various components of computer system 1100. Components includean input/output (I/O) component 1104 that processes a user action, suchas selecting keys from a keypad/keyboard, selecting one or more buttons,image, or links, and/or moving one or more images, etc., and sends acorresponding signal to bus 1102. I/O component 1104 may also include anoutput component, such as a display 1111 and a cursor control 1113 (suchas a keyboard, keypad, mouse, etc.). An optional audio input/outputcomponent 1105 may also be included to allow a user to use voice forinputting information by converting audio signals. Audio I/O component1105 may allow the user to hear audio. A transceiver or networkinterface 1106 transmits and receives signals between computer system1100 and other devices, such as another communication device, servicedevice, or a service provider server via network 140. In oneimplementation, the transmission is wireless, although othertransmission mediums and methods may also be suitable. One or moreprocessors 1112, which can be a micro-controller, digital signalprocessor (DSP), or other processing component, processes these varioussignals, such as for display on computer system 1100 or transmission toother devices via a communication link 1118. Processor(s) 1112 may alsocontrol transmission of information, such as cookies or IP addresses, toother devices.

Components of computer system 1100 also include a system memorycomponent 1114 (e.g., RAM), a static storage component 1116 (e.g., ROM),and/or a disk drive 1117. Computer system 1100 performs specificoperations by processor(s) 1112 and other components by executing one ormore sequences of instructions contained in system memory component1114. Logic may be encoded in a computer readable medium, which mayrefer to any medium that participates in providing instructions toprocessor(s) 1112 for execution. Such a medium may take many forms,including but not limited to, non-volatile media, volatile media, andtransmission media. In various implementations, non-volatile mediaincludes optical or magnetic disks, volatile media includes dynamicmemory, such as system memory component 1114, and transmission mediaincludes coaxial cables, copper wire, and fiber optics, including wiresthat comprise bus 1102. In one implementation, the logic is encoded innon-transitory computer readable medium. In one example, transmissionmedia may take the form of acoustic or light waves, such as thosegenerated during radio wave, optical, and infrared data communications.

Some common forms of computer readable media includes, for example,floppy disk, flexible disk, hard disk, magnetic tape, any other magneticmedium, CD-ROM, any other optical medium, punch cards, paper tape, anyother physical medium with patterns of holes, RAM, PROM, EEPROM,FLASH-EEPROM, any other memory chip or cartridge, or any other mediumfrom which a computer is adapted to read.

In various implementations of the present disclosure, execution ofinstruction sequences to practice the present disclosure may beperformed by computer system 1100. In various other implementations ofthe present disclosure, a plurality of computer systems 1100 coupled bycommunication link 1118 to the network (e.g., such as a LAN, WLAN, PTSN,and/or various other wired or wireless networks, includingtelecommunications, mobile, and cellular phone networks) may performinstruction sequences to practice the present disclosure in coordinationwith one another.

Where applicable, various implementations provided by the presentdisclosure may be implemented using hardware, software, or combinationsof hardware and software. Also, where applicable, the various hardwarecomponents and/or software components set forth herein may be combinedinto composite components that include software, hardware, and/or bothwithout departing from the spirit of the present disclosure. Whereapplicable, the various hardware components and/or software componentsset forth herein may be separated into sub-components that includesoftware, hardware, or both without departing from the scope of thepresent disclosure. In addition, where applicable, it is contemplatedthat software components may be implemented as hardware components andvice-versa.

Software, in accordance with the present disclosure, such as programcode and/or data, may be stored on one or more computer readablemediums. It is also contemplated that software identified herein may beimplemented using one or more general purpose or specific purposecomputers and/or computer systems, networked and/or otherwise. Whereapplicable, the ordering of various steps described herein may bechanged, combined into composite steps, and/or separated into sub-stepsto provide features described herein.

The various features and steps described herein may be implemented assystems that include one or more memories storing various informationdescribed herein and one or more processors coupled to the one or morememories and a network, wherein the one or more processors are operableto perform steps as described herein, as non-transitory machine-readablemedium that includes a plurality of machine-readable instructions which,when executed by one or more processors, are adapted to cause the one ormore processors to perform a method that includes steps describedherein, and methods performed by one or more devices, such as a hardwareprocessor, user device, server, and other devices described herein.

The foregoing disclosure is not intended to limit the present disclosureto the precise forms or particular fields of use disclosed. As such, itis contemplated that various alternate implementations and/ormodifications to the present disclosure, whether explicitly described orimplied herein, are possible in light of the disclosure. Having thusdescribed implementations of the present disclosure, persons of ordinaryskill in the art will recognize that changes may be made in form anddetail without departing from the scope of the present disclosure. Thus,the present disclosure is limited only by the claims.

What is claimed is:
 1. A method, comprising: receiving, by one or morehardware processors, voice data associated with a voice communicationbetween a user of a user device and a service provider server;determining, by the one or more hardware processors, that the voicecommunication is associated with a user account with the serviceprovider server; extracting, by the one or more hardware processors, aplurality of user attributes from the voice data; applying correspondingweights to one or more user attributes in the plurality of userattributes based at least in part on one or more characteristics of theuser account; determining, from the plurality of user attributes, aparticular user attribute combination; selecting, from a plurality ofvoice classifiers, a particular classifier corresponding to theparticular user attribute combination; generating, by the one or morehardware processors, a voice vector for the user based on the particularclassifier and the corresponding weights applied to the one or more userattributes; determining a difference between the voice vector and abaseline vector that corresponds to the user account; determining thatthe user is a malicious user when the difference exceeds a threshold;and in response to determining that the user is a malicious user,restricting the user from accessing the user account during the voicecommunication.
 2. The method of claim 1, wherein the comparing the voicevector to the baseline vector comprises: determining a deviation betweenone or more values in the voice vector and corresponding values in thebaseline vector; and determining whether the deviation between each ofthe one or more values and the corresponding values exceeds apredetermined distance threshold.
 3. The method of claim 1, furthercomprising: generating a notification indicating that the user is notauthenticated to access the user account; and transmitting thenotification to a communication device associated with the serviceprovider server.
 4. The method of claim 3, wherein the notificationprompts the communication device to prompt the user device foradditional verification information.
 5. The method of claim 1, furthercomprising: masking a subset of user attributes in the plurality of userattributes based at least in part on the one or more characteristics ofthe user account, wherein the generating the voice vector is furtherbased on the masking of the subset of user attributes, wherein themasked subset of user attributes is excluded from being consideredduring the determining the difference.
 6. The method of claim 1, furthercomprising determining the threshold based on the one or morecharacteristics of the user account.
 7. The method of claim 1, furthercomprising: obtaining a plurality of baseline vectors associated with aplurality of the user accounts; and comparing the voice vector to eachof the plurality of baseline vectors.
 8. The method of claim 7, whereinthe user account is a first user account, wherein the method furthercomprises: determining that the user was involved in a second voicecommunication associated with a second user account based on thecomparing the voice vector to each of the plurality of baseline vectors,wherein the determining that the user is a malicious user is furtherbased on the determining that the user was involved in the second voicecommunication.
 9. The method of claim 1, further comprising:determining, from the plurality of user attributes, a plurality of userattribute combinations; and generating a plurality of machinelearning-based networks based on the plurality of user attributecombinations, wherein the plurality of machine learning-based networkscorresponds to respective ones of the plurality of user attributecombinations.
 10. The method of claim 9, further comprising: trainingeach of the plurality of machine learning-based networks with a trainingdataset to form a plurality of machine learning-trained classifiers, thetraining dataset comprising attribute data that corresponds to one ofthe plurality of user attribute combinations, wherein the selecting theparticular classifier comprises selecting a machine learning-trainedclassifier from the plurality of machine learning-trained classifiers.11. The method of claim 1, wherein the particular user attributecombination comprises a gender-age-accent combination.
 12. A system,comprising: a non-transitory memory; and one or more hardware processorscoupled to the non-transitory memory and configured to executeinstructions from the non-transitory memory to cause the system toperform operations comprising: receiving user interaction dataassociated with an interaction between a user device and a serviceprovider server associated with a service, the user interaction datacomprising audio data associated with the interaction; extracting, usinga feature extraction engine, a plurality of user attributes from theaudio data; determining an intent of the interaction from the pluralityof user attributes; selecting, from a plurality of machinelearning-trained classifiers, one or more machine learning-trainedclassifiers that correspond to the determined intent; applying each ofthe one or more machine learning-trained classifiers to the plurality ofuser attributes; generating a plurality of voice vectors, wherein eachvoice vector of the plurality of voice vectors is associated with acorresponding one of the one or more machine learning-trainedclassifiers; selecting one of the one or more machine learning-trainedclassifiers that corresponds to one of the plurality of voice vectorshaving a highest aggregate score; determining a voice signatureassociated with the selected one of the one or more machinelearning-trained classifiers; determining whether the user deviceinteracted with the service provider server in a prior interaction basedon the voice signature; and sending a notification to a communicationdevice associated with the service provider server, the notificationcomprising an indication of whether the user device interacted with theservice provider server in the prior interaction.
 13. The system ofclaim 12, wherein the determining whether the user device interactedwith the service provider server comprises: accessing user accountinformation in a data repository communicably coupled to the serviceprovider server; comparing the voice signature to a historical voicesignature associated with the accessed user account information that isstored in the data repository; determining that a Euclidean distancebetween the voice signature and the historical voice signature is withina predetermined tolerance threshold; and determining that the userdevice interacted with the service provider server in the priorinteraction based on the Euclidean distance being within thepredetermined tolerance threshold.
 14. The system of claim 12, whereinthe determining whether the user device interacted with the serviceprovider server further comprises: obtaining a plurality of historicalvoice signatures from user account information associated withrespective ones of a plurality of user accounts; and comparing the voicesignature to each of the plurality of historical voice signatures. 15.The system of claim 12, wherein the operations further comprise:obtaining user account information from a user account associated withthe user device; determining, from the plurality of user attributes, aparticular user attribute combination based on the user accountinformation; selecting, from the plurality of machine learning-trainedclassifiers, a first machine learning-trained classifier thatcorresponds to the particular user attribute combination; applying thefirst machine learning-trained classifier to the plurality of userattributes; generating a voice vector comprising a plurality of scoresbased on the applying of machine learning-trained classifier, whereineach of the plurality of scores in the voice vector indicates alikelihood that an attribute of the plurality of user attributescorresponds to an attribute of a predetermined combination of features;comparing the voice vector to a baseline vector that corresponds to apredetermined combination of features; and sending a second notificationto the communication device associated with the service provider server,the second notification comprising a second indication of whether theuser device is verified based on the comparing.
 16. The system of claim12, wherein the operations further comprise: in response to determiningthat the user device interacted with the service provider server in theprior interaction, restricting the user device from accessing theservice during the interaction.
 17. A non-transitory machine-readablemedium having instructions stored thereon, the instructions executableto cause a machine to perform operations comprising: receiving userinteraction data associated with an interaction between a user deviceand a service provider server associated with a service, the userinteraction data comprising audio data associated with the interaction;extracting, using a feature extraction engine, a plurality of userattributes from the audio data; obtaining user account information froma user account associated with the user device; selecting a machinelearning-trained classifier that corresponds to a combination offeatures based on the user account information; applying the machinelearning-trained classifier to the plurality of user attributes;generating a voice vector comprising a plurality of scores based on theapplying of machine learning-trained classifier, wherein each of theplurality of scores in the voice vector indicates a likelihood that afirst respective attribute of the plurality of user attributescorresponds to a second respective attribute of the combination offeatures; comparing the voice vector to a baseline vector thatcorresponds to the combination of features; and sending a notificationto a communication device associated with the service provider server,the notification comprising an indication of whether the user device isverified based on the comparing.
 18. The non-transitory machine-readablemedium of claim 17, wherein the operations further comprise: determiningan intent of the interaction based on the plurality of user attributes;selecting one or more of a plurality of machine learning-trainedclassifiers that correspond to the determined intent; generating aplurality of voice vectors, wherein each voice vector from the pluralityof voice vectors is associated with a corresponding one of the one ormore of the plurality of machine learning-trained classifiers; selectingone of the plurality of machine learning-trained classifiers thatcorresponds to one of the plurality of voice vectors having a highestaggregate score; determining a voice signature that corresponds to theselected one of the plurality of machine learning-trained classifiers;determining whether the user device interacted with the service providerserver in a prior interaction based on the voice signature; and sendinga second notification to the communication device associated with theservice provider server, the notification comprising a second indicationof whether the user device interacted with the service provider serverin the prior interaction.
 19. The non-transitory machine-readable mediumof claim 17, wherein the operations further comprise: determining aplurality of user attribute combinations based on the plurality of userattributes; and generating a plurality of machine learning-basednetworks based on the plurality of user attribute combinations, whereinthe plurality of machine learning-based networks corresponds torespective ones of the plurality of user attribute combinations.
 20. Thenon-transitory machine-readable medium of claim 19, wherein theoperations further comprise: training each of the plurality of machinelearning-based networks with a training dataset to form a plurality ofmachine learning-trained classifiers, the training dataset comprisingattribute data that corresponds to one of the plurality of userattribute combinations, wherein the selecting the machinelearning-trained classifier comprises selecting the machinelearning-trained classifier from the plurality of machinelearning-trained classifiers.